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ABSTRACT 


The purpose of this research is to define and discuss the "best value" concept and how 
it relates to the procurement of engineering and technical services. The research was 
conducted by a review of regulations and policy guidance, interviews with Government 
and defense industry personnel, and visits to Navy Contracting Activities that procure 
engineering and technical services. The intent of the study was to analyze how the "best 
value" concept is currently implemented, problems with implementation, and to offer 
recommendations to improve best value procurements of engineering and technical 
services. This study also recommends areas for future study that may further improve the 


use of "best value" procurements for professional services. 
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I. INTRODUCTION 


A. AREA OF RESEARCH 

Achieving the "best value” contract is increasingly mentioned as an objective of 
Government procurement largely due to the many stories of poor performance attributed to 
contractors selected based on the low bid. As such, the "best value" concept is receiving 
much attention regarding its definition and implementation in source selection. Achieving 
the "best value" is based on reviewing all factors including price before making a 
procurement decision. The result should be the most advantageous procurement for the 
Government. 

Consumers utilize the "best value" concept on a daily basis when making value 
judgment tradeoffs between price and quality. Many consumers prefer higher quality over 
price while others prefer a lower price over quality. Depending on the consumer, either 
decision could equate to a "best value" judgment. The goal of "best value" in Government 
procurement is in determining the optimum tradeo!f between cost and quality representing 
the most advantageous procurement decision. 

"Best value" has applications extending to every competitively awarded contract for 
goods and services procured by the Department of Defense (DoD) and is not without its 
problems. In a system erected on the foundation of full and open competition, not 
awarding a contract to an offeror who is technically responsible and offering the lowest 
price could cause considerable protest, especially in the area of services which are 
traditionally more difficult to quantify. "Best value" issues revolve around areas including; 
proper acquisition planning for a "best value" procurement, the validity of technical and 


cost realism evaluations, and how to equate superiority of one contractor's technical 


approach with the lower bid of another contractor. From a contractor's standpoint, with a 





decreasing number of contracts available, there exists a high level of distrust in the "best 
value” source selection process. Many contractors believe that, no matter what the 
Government says, the competition is based on low bid. This thesis research will examine 
the "best value" concept in source selection for engineering and technical services contracts. 
The objective is to analyze the current use of the "best value" concept and make 
recommendations for improvement of the process. 

This research was accomplished through personal and telephone interviews with 
Government and defense contractor personnel. In addition to the interviews, examples of 


procurement documentation were collected and analyzed. 


B. RESEARCH QUESTIONS 
1. Primary Research Question 


What is the most efficient and effective method of implementing "best value" in 
source selection for engineering and technical services contracts? 


2. Subsidiary Research Questions 


What is "best value"? 

Under what circumstances is "best value" employed? 

What are the "best value" source selection techniques for engineering and technical 
services? 

What factors and subfactors are utilized in "best value” source selections for 
engineering and technical service contracts? 

What are the problems or impediments to implementing "best value" source 
selection for engineering and technical services? 

How do contracting activities quantify paying a premium on "best value" 
procurements? 


Ce a, ee 


C. OBJECTIVE 

The objective of this research is to examine the intricacies and problems associated 
with the use of the "best value” concept in source selection. By doing so, qualitative 
insight will be provided to procurement personnel and engincering support services 


contractors. 


to 








D. SCOPE OF THESIS 


The scope of this thesis revolved around the acquisition planning and evaluation 
phases of the contracting cycle for engineering and technical services contracts. Contract 
administration is only briefly mentioned in the context of viewpoints collected during 
research on a specific problem; contractor gaming, which is discussed in Chapter IV. The 
research focused on Navy contracting activities although other DoD agencies and the 
National Aeronautics and Space Administration (NASA) were contacted for their positions 
in several areas. By contacting other agencies, the researcher was able to compare 
problems and solutions to the "best value" approach. Similarities between systems 


commands and field activities were expected. 


E. METHODOLOGY 

The methodology used in this study consisted of a comprehensive literature review, 
personal, and telephone interviews, and onsite visits to contracting activiues. This 
methodology was necessary because the "best value" concept is somewhat nebulous, and 
as a result, interpersonal communications were required. A sample of eight to ten 
engineering services Contracts was reviewed in the study. Government acquisition 
personnel, defense contractors, and requiring activities were targeted as the sources of 


information for the study. 


F. ORGANIZATION 
Chapter I defined the research issue and its importance. The objective, scope, and 


methodology of the thesis were also presented. 


Chapter II] examines a definition of the "best value" concept presented in the literature 
and discusses when “best value" should be utilized. In addition, the chapter defines 
engineering and technical services and "best value" source selection methods used to obtain 
these services. 

Chapter III examines responses received from a survey of Government and contractor 
personnel. In addition, data obtained from a sample of engineering and technical services 
contracts is presented. 

Chapter IV presents an analysis of problems collected from the interviews with 
Government and contractor personnel. 

Chapter V provides conclusions and recommendations germanc to the research 


conducted. Additionally, the research questions are answered and areas for further 


research are presented. 








II. BACKGROUND 


A. INTRODUCTION 

This chapter will define the "best value" concept and discuss when "best value" is 
utilized. This chapter also introduces a definition of engineering and technical services, and 
a discussion of "best value" source selection methods used to procure engineering and 


technical! services. 


B. "BEST VALUE" DEFINED 

Examples of definitions for "best value" from a literature search are presented below’. 

In a report on expanding the use of commercial components in military equipment, the 
Defense Science Board offered the following efinition: 

"Best value"— the evaluation of a (commercial) product, system or service based on 
all reasonable factors including but not limited to initial price, life cycle costs, available 
extended warranties, prior product experience, availability of distribution and service 
channels, past producer performance, past vendor performance and so forth, for the 
purpose of procuring a product, system or service that provides optimum satisfaction of 
the mission needs. [Ref. 1:p. 81] 

The Naval Supply Systems Command offers a similar definition by describing "best 
value” as a process which, "Describes all those acquisitions where selection for award is to 
be based on other than the lowest evaluated price or lowest cost/price among technically 
acceptable offers." [Ref. 2:p. A-4] 

The Naval Air Systems Command (NAVAIR) is more succinct with its definition of 
"best value" which encompasses not only the process of proposal evaluation on factors 
other than price, but also incorporates the definition of the requirement as well. In a 1990 


Business Strategy Presentation, NAVAIR defined "best value” as "obtaining the right 


product at the nght price, and not making an acquisition decision on price alone." 








The Defense Logistic Agency (DLA), in a handbook for contracting officers, DLA 
Handbook 4105.3 [Ref. 3], is more expansive in its definition of "best valuc". It considers 
the usage of value judgments for all methods of source selection. These value judgments 
are employed as methods of achieving the most advantageous procurement decision. For 
instance, in a source selection based on low price, the contracting officer should consider 
additional costs to the Government of quality deficiency reports and delinquency on past 
contracts incurred by the offeror being evaluated. These added costs should be factored in 
as expected costs prior to making an award decision. Buying the "best value” in this 
context refers to awarding a contract on the low overall cost instead of low initial price. 
DLAH 4105.3 recommends techniques to obtain the objective of "best value" for various 
methods of source selection including formal source selection, streamlined source 
selection, responsibility determinations, quality vendors, and multi-source contracting. 

The Defense Personnel Support Center (DPSC) in a handbook or. source selection, 
DPSC Handbook 4105.8, describes "best value" as a source selection technique which 
equates to a streamlined version of formal source selection. "Best value" source selection 
in this context, is utilized for less complex procurements with dollar values under $1 
million [Ref. 4:p. 7]. 

The Defense Analysis and Studies Office also viewed "best valuc" as a variation on 
source selection when it reported: 

One variation on source selection, "Best Value" Source Selection (BVSS), 
includes the same elements used in the usual source selection process. It is 
streamlined, however, by lowering the approval signature levels and by simplifying 
some of the procedures. BVSS also places emphasis on contractor past performance in 
both quality and on time delivery. (Ref. 5:p. 5] 

The Air Force, Army, and NASA utilize source selection methods based on factors 


other than price alone, yet the term "best value”, "best buy" or "greatest value" could not be 
Pp ) ) & 


located in their source selection literature. General references allude to source selection 








methods which “allow an award to be based on factors other than price" [Ref. 6:p. 2] and 
"selecting the source whose proposal presents the highest probability of quality 
performance to Lest meet NASA's requirements at a realistic cost/price" [Ref. 7:p. i]. 
However, none mention “best value" as a specific type of procurement process. 
References to "best value", greatest value, and greatest overall benefit to the Government 
are made in sample Request For Proposals (RFP)s from all DoD agencies, further 
indicating that all agencies mentioned do have an appreciation for the concept. 

For purposes of this research, a working definition for "best valuc" is presented in 
terms of an objective and a process. As an objective, "best value" represents the most 
advantageous procurement decision available to the Government. This decision is based on 
low price, low overall cost or other factors. "Best value" defined as an objective has 
extremely broad applications. One can obtain the "best value" under any source selection 
method as long as the product or service meets the criterion of providing optimum 
satisfaction of mission needs. 

"Best value" is also defined in terms of an evaluation process where all relevant factors 
are taken into account prior to making a procurement decision. The end result of this 
evaluation process represents the most advantageous procurement decision. In this 
context, procurement decisions are not based on price alone. 

As an evaluation process, "best value" directly contrasts with technically acceptable 
low offeror evaluations which make contract awards based on the low price among 
technically acceptable offerors. “Best value" emphasizes other factors in addition to price 
and establishes different levels of technical merit from which to make the most 


advantageous procurement decision. 





C. FAR AND DFARS GUIDANCE ON "BEST VALUE" 


A review of the FAR and DFARS for the term "best (greatest) value" uncovers very 
few specifics, although factors that imply the use of the "best value" concept of making the 
most advantageous procurement decision based on price and other factors are implicit 
throughout. 

The only section of FAR within which the "best value" concept is specifically referred 
to is in the source selection subpart 15.605, which states: 

Where the lowest price or lowest cost to the Government is properly the deciding 
factor in many source selections, in certain acquisitions, the Government may select the 
source whose proposal offers the greatest value to the Government in terms of 
performance and other factors. [Ref. 8:p. 15-20] 

The FAR continues its reference to "best value" as follows: 

In awarding a cost reimbursement contract, the cost proposal should not be 
controlling since advance estimates of costs may not be valid indicators of actual costs. 
The primary consideration should be, which offeror can perform the contract in a 
manner most advantageous to the Government as determined by evaluation of 
proposals according to the established evaluation criteria. [Ref. 8:p. 15-20] 

A recent change to the DFARS, published in the Federal Register, Vol. 56, No. 172, 
page 43986, on September 5, 1991, is more explicit with regard to "best value" in the area 
of services. The change is a result of Section 834 of the 1991 National Defense 
Authorization Act which requires DoD, to the maximum extent possible, to acquire services 
on the basis of tasks performed rather than on the basis of number of hours the service is to 
be provided. DFARS section 215.605 is amended to read as follows: 

215.605 Evaluation Factors 


(C) In Competitive acquisitions of services— 


(1) Evaluation and award should be based to the maximum extent practicable, on best 
overall value to the Government in terms of quality and other factors. 


(2) The weighting of costs must be commensurate with the nature of the services 
provided 








The change continues by delineating when it might be appropriate to use low price 
technically acceptable offeror, and when it might be more appropriate to award to an offeror 
based on technical and quality considerations at other than the lowest price. The DFARS 
change also adds section 237.170 which provides guidance and prescribes a new 


solicitation provision and contract clause on uncompensated overtime. 


D. WHEN IS "BEST VALUE" UTILIZED? 

The use of "best value" is generally reserved for the procurement of complex goods or 
services where alternate approaches for solving a requirement are available. This is due to 
the often times subjective nature of a "best value" procurement where evaluation factors and 
criteria within those factors are utilized to discriminate between solutions offered by 
competing proposals. As a result, requirements necessitating the use of performance 
specifications or complex services are typically considered for "best value" evaluations. 
Conversely, products with detailed specifications and routine services are generally 
acquired based on low cost. 
| In a memorandum to program executive officers, direct reporting program managers, 
commanders of system commands, and others, the Assistant Secretary of the Navy for 


wy 


Research, Development and Acquisition, the Honorable Gerald Cann described using "oest 


value" as follows: 


"Best Value" evaluations should be employed whenever possible and especially in 
weapons system, federal information processing and professional and technical service 
support acquisitions. Requirements which dictate complex integration of people, 
equipment, hardware, innovation and software should also be evaluated on a "best 
value” basis. However, when non-complex, routine requirements are being procured 
and the product to be delivered is clearly defined at the outset of the procurement, it 
may be appropriate to award to the lowest priced, technically acceptable offeror. [Ref. 
9, Appendix A] 





Duc to the non-definitive nature of "best value" procurements, contract type selection 


is normally cost reimbursement. This is not to say that fixed price arrangements are not 
used for "best value" procurements. However, 

Great care is required in the application of greatest (best) value to fixed price efforts 
since the application of evaluation factors to distinguish between offers for an item 
which has an adequately detailed specification may involve applying responsibility 
factors rather than technical acceptability factors. [Ref. 2:p. A-4] 

The point is that the contracting activity cannot eliminate an offeror, especially a small 
business for responsibility type issues through the use of evaluation factors. The 
responsibility determination 1s made after the "best valuc" offeror is selected. 

Comptroller General decision B-238953.4 of 28 September 1990, in the matter of 
Flight International Group, Inc. (FLIGHT), may be instructive concerning the issue of 
responsibility determination in lieu of "best value" evaluations. In this case, the Navy used 
a responsibility factor to eliminate a small business under the guise of a "best valuc” 
evaluation. 

FLIGHT challenged the Navy's evaluation of its proposal because the financial 
condition of the company was not stated as an evaluation factor in the solicitation. The 
solicitation requested proposals for a firm fixed price contract to provide flight training 
services for Naval Flight Officers for one base year and four option years. The solicitation 
provided for an award to the offeror whose proposal presented the "best value" to the 
Government on the basis of three factors; technical approach, and price, which were 
weighted equally, and management, which was considerably less important. During 
proposal evaluation, it was learned through a newspaper article that FLIGHT, a small 
business, was having financial difficulties and was “considering options to meet liquidity 
problems." The contracting officer contacted the cognizant Defense Contract 


Administration Services Management Area (DCASMA) which recommended against the 


award. The contracting officer then failed to contact the Small Business Administration for 


10 








a required certificate of competency determination for FLIGHT. This was duc to the 
pressure of a time constraint to award the contract. The result was an evaluation of 
FLIGHT's proposal: 
On the basis of inadequate financial] capacity but under the guisc of a comparative 
"best value" evaluation (which) effectively constituted a finding of nonresponsibility 
which the Agency was required to refer to the Small Business Administration. [Ref. 
10:p. 2) 

The protest by FLIGHT was sustained. The Navy was ordered to terminate the 
current contract for convenience and award the contract to FLIGHT. In the FLIGHT casc, 
the Navy employed a responsibility type factor to eliminate a small business in lieu of a 
certificate of competency determination from the Small Business Administration. 

E. WHAT ARE ENGINEERING AND TECHNICAL SERVICES AND 

WHEN ARE THEY USED? 

Service contracting characteristics and governing regulations are described in FAR part 
37 which defines a service contract in subpart 37.101 as: 

A contract that directly engages the time and effort of a contractor whose primary 
purpose is to perform an identifiable task rather than furnish an end item of supply. A 
' service contract may be a non-personal or personal contract. It can also cover services 
performed by either professional or non-professional personnel whether on an 
individual or organizational basis. [Ref. 8:p. 37-1] 
The DoD awards service contracts for efforts encompassing routine, recurring type services 
to sophisticated research and design efforts. 

As delineated in the FAR, part 37, engineering and technical services are one type of 
Contracted Advisory and Assistance Services (CAAS) which take the form of 
"Information, advice, opinions, alternatives, conclusions, recommendations, training or 
direct assistance" [Ref. 8:p. 37-3]. Besides engineering and technical services, other 


services covered under the CAAS genre include individual experts and consultants, studies, 


analyses and evaluations, and management and professional support services. 








Contractor engineering and technical services are defined in DFARS 237.701 as: 

The furnishing of advice, instruction, and training to DoD personnel, by 
commercial or industrial companies in the installation, operation, and maintenance of 
DoD weapons, equipment and systems. This includes transmitting the knowledge 
necessary to develop among those DoD personnel the technica! skill required for 
installing, maintaining and operating such equipment in a high state of military 
readiness. [Ref. 11:p. 237-6] 

DFARS further separates engineering and technical services into three categories which 
include contract field services, contract plant services, and field service representatives. 

1. Contracting for Services 

Service contracting is differentiated from hardware procurement in that with 
services, the Government is acquiring expertise, knowledge, and time instead of an end 
item of supply. As a result, the process of acquiring services is somewhat different. 

In acquiring services, the requiring activity must first make a determination that the 
services cannot be performed "in house"; meaning Government employees have neither the 
expertise nor the resources to perform the requirement. Once this determination is made, 
the requiring activity constructs a Performance Work Statement (PWS) in heu of a 
specification for hardware requirements. The PWS is a document which describes the 
essential and technical requirements for the service and the standards used to determine 
whether the requirements are met. The PWS for engineering and technical service contracts 
requiring "best value" source selection is usually broad in scope providing the flexibility 
necessary to meet the needs of dynamic programs. This is in contrast to well defined 
PWSs for routine services and design specifications for hardware procurements. 

Contracting for services employs a different set of evaluation criteria than 
contracting for hardware and other tangible items. Since technical expertise is of 
paramount importance with services, technical factors are heavily weighted. These include 


the contractor's technical approach to solving the requirement, the level of experience 


among key personnel and corporate expenence in similar or past efforts. 





Solicitations for engineering and technical service contracts will normally include 


an estimated ceiling covering the number of hours during a base year and subsequent 
option years, and the labor categories, or “labor mix," necessary to complete the tasks 
presented in the PWS. Categories in the labor mix are sometimes equated to an equivalent 
Government service rating, supplying the offeror with the minimum pay scale expected. 
These rates are then used as benchmarks to evaluate the cost realism of the offeror's 
proposal. The cost realism analysis is discussed further in the next section of this chapter. 

Engineering and technical services are procured by field contracting activities, 
laboratones, weapon testing sites, training centers and systems commands. Examples of 
engineering and technical services encountered by this researcher include configuration 
management support, systems and software engineering support, services in support of 
weapons lethality assessment, and cruise missile technical support services among others. 

Contracts for engineering and technical services usually consist of one base year 
with the option to continue services for up to four additional years. Therefore, if all 
options are exercised, the maximum length of a contract can extend to five years prior to 
resoliciting the requirement. The contracts can be omnibus in nature, covering many arcas 
of support, or more specific, requiring support for only one area. 

The typical contract types for engineering and technical services encountered by 
this researcher were 1) Cost Plus Fixed Fee (CPFF), 2) Time and Materials (T&M), 3) 
Indefinite Delivery Indefinite Quantity (IDIQ), and 4) Cost Plus Award Fee (CPAF). As 
an observation, activities used either the CPFF and CPAF or T&M and IDIQ. Only one of 
the activities encountered during this research used a mixture of Cost Plus and T&M type 


contracts. 





F. DESCRIPTION OF "BEST VALUE" SOURCE SELECTION FOR 
ENGINEERING AND TECHNICAL SERVICES 


As discussed in Section A, "best valuc" is generally defined in terms of a source 
selection evaluation process that is normally reserved for non-routine service requirements. 
"Best value" source selection employs evaluation factors to discriminate between offerors. 
The evaluation can be either formal or streamlined. Both techniques, as applied to 
engineering and technical services, are described below after a brief discussion of 
acquisition planning. 

1. Acquisition Planning 

A key in "best value" source selection is acquisition planning. Acquisition 
planning considers elements including a clear definition of the requirement, objectives of 
the procurement, and establishes the composition of the acquisition team. Each element is 
critical to the success of the procurement. The products of acquisition planning include a 
well defined PWS and a Source Selection Plan (SSP). Although not always required, and 
depending on the designation of the program or dollar value of the acquisition, the 
advanced planning required to effectively select a source dictates the need for a written 
acquisition plan. 

A typical SSP consists of two parts. Part One will provide an overview of the 
acquisition strategy and describe the organization, membership, and responsibilities of the 
source selection .eam. Part Two identifies the evaluation criteria and the detailed 
procedures for proposal evaluation [Ref. 12:p. 16]. It is of utmost importance to 
thoroughly plan the evaluation criteria to be used in evaluating proposals. Developing the 
appropniate evaluation criteria is perhaps the most important aspect of acquisition planning 
and is considered to be the most difficult. 

Part Two of the SSP also contains the procedures to be implemented for evaluating 


contractor proposals. The topics to be covered will depend on the size and complexity of 


14 





the acquisition, and may include; 1) the nature and scope of the evaluation, 2) preparation 
and training of the personnel involved, 3) evaluation committees, 4) scoping techniques, 
5) risk assessment, 6) price or cost evaluation, and 7) documenting the evaluation. [Ref. 
12:Appendix B]} 

Upon completion of the SSP, it is necessary to solicit proposals from industry. 
This action is initiated through the RFP. All sections of the RFP are important to the 
successful evaluation of proposals. However, Sections B, C, L, and M are especially 
vital. Sections B and C, "PWS/Specifications," contain the Government's requirement. It 
is essential to have a clear and concise defimition of the requirement. Section L, 
"Instructions, Conditions, and Notices to Offerors or Quoters," is used to deliver 
instructions to offerors on how to prepare and deliver this proposal. Section M, 
"Evaluation Factors for Award," must contain the following information in accordance with 
FAR subpart 15.605 (E); 

The solicitation shall clearly state the evaluation factors, including cost or price and 
any significant subfactors, that will be considered in making the source selection and 
their relative importance. Numerical weights, which may be employed in the evaluation 
of proposals, need not be disclosed in solicitations. The solicitation shall inform 
offerors of minimum requirements that apply to particular evaluation factors and 
significant subfactors. [Ref. 8:p. 15-20] 

Additional steps can be taken to ensure the RFP is as complete as possible. These 
steps include the utilization of a draft RFP or presolicitation conference which allow 
industry the opportunity to carefully review the requirement and provide feedback on 
possible problem areas. 

2. Formal Source Selection Method 


The establishment of a specific evaluation and decision organization is what 


designates certain source selections as formal. The formal source selection method is a 
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time consuming and administratively expensive process generally reserved for major 
weapon systems and other complex requirements. It can also provide the greatest 
flexibility to DoD in selecting the most advantageous contractor proposal. 

Responsibilities of the formal source selection organization from DLA Handbook 
4105.3 are listed below: 

1. The Source Selection Authonty (SSA) is the individual responsible for making the 
final decision on recommendations from the Source Selection Advisory Council 
(SSAC) and the Source Selection Evaluation Board (SSEB). The SSA is normally 
a high ranking official in the organization possessing a broad perspective of the 


organization's mission and is in the best position to make a decision based on the 
agency and its requirements. 


Ww 


. The SSAC assists the SSA and SSEB in an advisory status. It reviews and 
approves factors and their weighting used in the source selection evaluation and 
appoints chairpersons and members of the SSEB. 


3. The SSEB is responsible for the development of evaluation standards, evaluation of 
proposals in accordance with those standards and presentation of findings to the 
SSA and SSAC. The standards relate to technical and cost factors for each 
proposal. The purpose of the SSEB is an agreement on the strengths and 
weaknesses of each proposal against the SSP. These strengths and weaknesses are 
presented to the SSA as an individual proposals. Proposals are not compared by 
the SSEB. 


4. The contracting officer is responsible for preparing the acquisition plan, preparing 
the solicitation, evaluating proposal costs to determine its reasonableness and 
making the award. The contracting officer is normally assigned as the chairperson 
of the SSEB cost section which evaluates proposal costs for reasonableness and 
realism. (Ref. 3:p. II-3] 

a. Evaluation Factors 
The process of formal source selection facilitates an agency's ability to make 
the most advantageous procurement decision through the use of evaluation factors. 
Evaluation factors are used to discriminate between offerors. A procuring agency has great 
latitude in determining which factors should be utilized. Three requirements on the 


selection of evaluation factors from DLA Handbook 4105.3 are: 


1. The factor must be variabic, i.e., there must be a reasonable expectation of variance 
between offerors. 
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2. The variance must be measurable. This does not mean it must be quantifiable; 
qualitative measures are equally valid. 


3. The factor must be determinant. The Comptroller General has stated in several 
decisions that use of an area as an evaluation factor is valid only if the agency's 
needs warrant a comparative evaluation of those areas. The FAR reinforces this by 
Stating that the evaluation factors will include only those factors that have an impact 
on the source selection decision. The simplest way to assess determinance is to ask 
yourself: “is the Government willing to pay more for higher merit in this factor?" 
{Ref. 3:p. II-5] 

Typical factors utilized in the source selection process for engineering and 
technical services include technical, management and cost. In accordance with FAR part 
15.605(b), cost or price and quality are mandatory evaluation factors in all procurements. 
Quality is normally considered in the context of all factors including the technical 
excellence, management capability or past performance of an offeror. Cost is listed in 
Section M as a separate evaluation factor. 

In order to facilitate the evaluation process, factors are normally broken down 
into subfactors. An example of a technical factor broken down into subfactors from an 
RFP by the Naval Regional Contracting Center (NRCC), San Diego, Long Beach 
Detachment, is presented below: 

Factor: Techni al 

Subfactors: a) Technical Approach 

b) Corporate Experience 
c) Personnel Qualifications 
d) Management Plan 

e) Quality Assurance Plan 

Factors and subfactors are weighted in accordance with their relative 
importance. These weights are communicated to offerors via the solicitation. This can be 
accomplished by listing factors and significant subfactors in their relative order of 
importance, or by using adjective descriptions of the factor's importance, 1.e., "Technical is 


more important than management. Technical and management factors combined are 


slightly more important than cost.” 


17 


Factors are also designated as variable in relative importance. For instance, if 
the difference between two offerors 1s insignificant on the most important factor, the 
difference in lesser factors may become more important in the overall evaluation. 

In evaluating factors and subfactors, the SSEB utilizes evaluation criteria such 
as 1) the soundness of the technical approach, 2) compliance with the requirements, 3) 
excellence of designs, and 4) demonstrated understanding of the requirement [Ref. 3:p. 
I]-8]. Multiple assessment criteria can result in a matrix evaluation arrangement which can 
assist in visualizing the overall proposal evaluation. 

b. Rating Systems 

Rating descriptions of each factor could be based on point systems, adjective 
grades, colors, or a combination, depending on the agency and activity within the agency. 
For example, the Air Force is required by regulation to use colors as a rating system, 
whereas Navy activities contacted by this researcher have used points, adjectives, a 
composite system of points and adjectives, and colors. Each system, by its nature, has 
advantages and disadvantages. However, if used properly as a tool of the SSA, the results 
should be the selection of the "best value" offeror. 

(1) Point Systems. Of al] the rating systems available, point systems are the 
most precise. They are typically based on a 100 or 1000 point scale. Points are 
accumulated during proposal evaluation in the form of numerical scores, which allow the 
SSA to make total point comparisons between competing proposals. For example, one 
offeror may achieve a total score of 75 points on its proposal compared to a superior 
proposal which achieved a score of 85 points. Point systems do not allow the flexibility of 
adjective ratings and colors in making tradeoffs between evaluation factors. This is due to 
the precise grades given to proposals where the differences are sometimes difficult to 


equate with a precise numerical score. 
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(2) Composite Rating Systems. Composite rating systems consist of 
numerical scores attached to an adjectival or color descriptor. Composite systems utilize a 
numeric scale and assign an adjectival or color descriptor depending upon where the total 
points achieved by the proposal fall on the scale. For example, on a numeric scale of 0 
100, any proposal achieving a numeric score between 90-100 total points will be assigned 
an adjectival descriptor of "outstanding" and a color descriptor of "blue". Composite rating 
systems eliminate the need to differentiate when proposals are only two or three points 
apart under a point system. However, there may be a considerable difference between 
proposals on the high and low ends of the adjectival grade of "outstanding". 

(3) Adjectival and Color Systems. Employing strictly adjectival grades or 
colors as a rating system permits the SSA the most flexibility in making contract awards 
because it does not constrain the SSA to making awards based on high point scores 
associated with point and composite systems. In fact, using strictly adjectives or colors 
requires the SSA to focus more on concrete differences in proposals rather than relying on 
point scores. A disadvantage of adjectives and colors is the lack of preciseness sometimes 
desired by the technical codes evaluating an offeror's proposal. As an example of rating 
systems, consider Table 1: 


Table I 


P| score | scale | adjective Rating | Color Rating 
oewra | 92 | 50-100 | outsanding | Bive 
lore | ss | sos 


The difference between Offeror A and Offeror B equals only four points in 






numerical scores, yet equates to an adjectival rating difference between an “excellent” and 
"outstanding" offeror. If the numerical grades were 95 for Offeror A and 85 for Offeror B, 


the adjective rating is more descriptive. 





The Naval Supply Systems Command (NAVSUP) prefers usage of the 
composite method but recognizes that: 

A variety of (rating systems) methodologies have been advocated and successfully 
employed throughout the Navy Field Contracting System (NFCS) which offer different 
advantages and drawbacks. . .activities are authorized to employ the system which 
appears most advantageous to their needs. [Ref. 2: p. 11-12] 

DLA also recognizes a variety of rating systems and stresses systems 
which: 


.. allow the SSA the maximum flexibility in making tradeoffs between evaluation 
factors. Numerical systems present the most problems in this area. [Ref. 3:p. 8-11] 


Unlike NAVSUP and DLA, the Air Force mandates the use of a color 
coded rating system. 

Use colors to indicate proposal ratings at the factor and subfactor level. At the area 
and item level, color codes shall also be used to depict the rating and will be 
accompanied by a consistent narrative assessment detailing the basis for the rating. 
{Ref. 6:p. 11] 

Regardless of the rating descnption utilized during the source selection, it 
is the responsibility of the SSA to make the source selection decision based on the relative 
strengths and weaknesses of each proposal. Rating descriptions are a tool of the SSA. 
Two proposals evaluated as equal overall under a point system, adjectives, or colors may 
be considerably different upon examining the discriminators between the offerors. If two 
competing proposals are evaluated equal overall, one of the key points is to look back at the 
RFP and the SSP and determine which factor was considered most important. It may 
happen where one offeror has done very well on factors not considered the most important 
and overall, achieved an equal grade with the offeror who did well on the factor considered 


most important. This requires an investigation before awarding the contract and is why 


narrative comments from the technical evaluation board are crucial. 


The bottom line in designing a raling system is to maximize useful 
information to the SSA without requiring an extensive review of each proposal by the SSA. 
This is achieved through a proper design and evaluation of factors by the SSEB and a 
presentation of each contractor's strengths and weaknesses to the SSA. 

c. Cost Realism Analysis 
The cost realism analysis is completed by the cost pancl of the SSEB in 

conjunction with the technical panel. It is used in the evaluation of cost reimbursement type 
contracts even when adequate price competition exists. A cost realism analysis is generally 
not used when fixed price and time and materials type contracts are contemplated because 
prices are fixed and there is no opportunity for an upward adjustment in the cost proposal. 
However, like all contracting actions, fixed piice and time and materials type contracts do 
require a reasonableness evaluation. As defined in the DFARS, Section 215.801, the cost 
realism analysis is: 

A review of the cost position of an offeror's proposal to determine: 

1. If the overall costs proposed are realistic for the work to be performed; 


2. If the costs reflect an offeror's understanding of the requirements of the 
proposed contractor; 


3. If the costs are consistent with the various elements of the technical proposal. 
[Ref. 11: p. 215-2} 


For example, if an offeror neglects certain costs required to complete the technical 
requirements, it may indicate an incomplete or poor understanding of those technical 
requirements and an increased risk to the Government. A cost realism analysis is also 
conducted for use in the cost/technical tradeoff analysis if required prior to contract award. 
In this respect, the cost realism analysis evaluates the contractor's costs to determine their 
reliability. If the proposed costs are found to be inaccurate, the costs are adjusted. For 


example, if an offeror proposes unrealistically low labor rates in a Best and Final Offer 





(BAFO) without justification, costs would be adjusted upward to reflect accurate costs in 
accordance with the cost realism assessment. The upward adjusted cost is used in 
conjunction with the technical evaluation in the overall proposal assessment. 
d. Cost Factor 

The cost factor for evaluation is weighted, although it is never scored in "best 
value" evaluations. Weighting costs means an assignment is made indicating the relative 
importance of cost in comparison to the technical proposal. For instance, technical and 
management factors combined may be slightly more important than cost. If translated to 
points, this may equate to a 60/40 split which means that out of 100 total points, 60 are 
attributed to technical and management factors, while 40 are accounted for in cost. Only a 
few NAVSUP activities were observed during this research to use a point breakdown of 
cost to technical factors. All other Navy activities, the Air Force, Army and NASA claimed 
to evaluate the cost factor by assigning a relative, not a numerically specific breakdown. 
Evaluating cost against other factors is thereby more flexible and less dependent on a 
mathematical formulation. 

The scoring of the cost factor, which is not permitted by any DoD agency or 
NASA, is the assignment of points, adjective grades, or color ratings to the accuracy of 
costs in the proposal. The reason this practice is prohibited is because scoring can skew an 
evaluation toward a contractor with accurate costs but a marginal technical proposal. Even 
more significant, is the possibility of making a technically unacceptable proposal acceptable 
for award based on the accuracy of costs alone. For example, if an offeror's costs are 
determined to be accurate through a cost realism analysis, that offeror would receive 
additional points even though the technical proposal was marginal or unacceptable. The 
additional points for cost accuracy may make a marginal or unacceptable proposal eligible 


for contract award. 
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e. Competitive Range Determination 

Upon completion of the initia! evaluation phase and before discussions, a 
competitive range determination is made [Ref. 3:p. II-16]. This determination is made by 
the contracting officer with input by the technical panel. The determination is particularly 
important in a formal source selection where costs associated with proposal preparation and 
evaluation are great. 

In determining the competitive range, a tradeoff between a broad competitive 
base and the opportunity of award requires consideration. In such a case, it is generally 
considered prudent to err in favor of a broader range. FAR subpart 15.609(a) states that 
"when there is doubt as to whether a proposal is in the competitive range, the proposal 
should be included." [Ref. 8:p. 15-22 

Competitive range determinations are made based on all factors in the 
solicitation and in the same manner a proposal is evaluated for contract award. Competitive 
ranges are established on the basis of comparative assessments of proposals in which an 
average proposal may be rejected based upon the outstanding merits of others. As 
previously noted, contracting officers must be careful not to eliminate small businesses 
based on responsibility factors when a certificate of competency determination is required. 
Instead, a comparative assessment based on the merits of other proposals should be 
employed when eliminating any proposal from the competitive range. 

f. Discussions 

The next stage of formal source selection is called discussions. Discussions 
are defined in the FAR part 15.601 to include: 

. . any written or oral communication between the Government and the offeror that 


involves information essential for determination of acceptability of a proposal or that 
provide the offeror to revise or modify its proposal. [Ref. 8:p. 15-19] 





Apparent clerical mistakes may be corrected prior to the establishment of a competitive 
range and without entering into discussions where they qualify as clarifications. 

There is a fine line between clarification and discussions, and as a result, all 
correspondence between the evaluation board and the offeror should be first directed 
through the contracting officer [Ref. 3:p. II-18]. If responses to clarification requests 
represent modifications or revisions to an offeror's proposal, those modifications or 
revisions should not be forwarded to the evaluation board since it may change the 
evaluation in favor of the offeror making the change. The changes could possibly add an 
offeror to the competitive range who was previously not considered susceptible for award. 
Clanfications for offerors already in the competitive range can be deferred until discussions 
have commenced. 

Discussions are conducted by the contracting officer with technical evaluators 
present as requested. The idea is to clear up questions regarding the offeror's proposal and 
inform the offeror of deficient areas prior to BAFOs. Extreme care must be taken during 
discussions so as not to “technically level" the offerors by providing solutions to 
deficiencies in the proposals noted during the evaluation [Ref. 3:p. I]-18]. Another practice 
to avoid is that of technical transfusion, where one offeror's superior technical solution is 
communicated to lower rated technical offerors [Ref. 3:p. II]-18]. The point is to avoid 
discussing solutions to the deficiencies, and instead communicate why the evaluation team 
determines certain areas of the proposal as deficient. 

Cost discussions permit the contracting officer to question offerors on the cost 
elements of their proposals. For instance, a cost proposal may lack certain cost elements, 
or the cost proposed may be unrealistic in view of the level of effort expected. Both 
deficiencies may be addressed during discussions. This allows the offeror the opportunity 


to reevaluate its position for possible revision in the BAFO [Ref. 3:p. I-18]. 
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Agencies were recently empowered with the opportunity to make awards 
without discussions to other than the low price offeror provided the cost 1s competitive and 
the competing offeror is not susceptible for award. The revision is a result of Section 802 
of the 1991 National Defense Authorization Act and incorporated in FAR 15.610 [Ref. 8:p. 
15-22}. Past policy required awards without discussions be made to the offeror with the 
low price bid. Awarding without discussions saves the contracting agency and offerors 
time and resources associated with conducting discussions and subsequent BAFOs. 

g. Best and Final Offers (BAFO) 

BAFOs provide all offerors in the competitive range with a common cutoff date 
to submit modifications to their proposals, ideally directed at the deficiencies noted during 
discussions. There is no limit on the changes an offeror can make during the BAFO stage 
although it is possible to require offerors to illuminate changes for ease of evaluation. 
[Ref. 3:p. I]-19] 

After receipt of BAFOs, all offers in the competitive range are evaluated once 
more in accordance with the factors and evaluation criteria listed in the SSP. The only 
difference is now the evaluation should focus on deficiencies noted during discussions and 
addressed in the BAFOs. At completion, the strengths and weaknesses of each proposal 
should be presented to the SSA who makes the final decision that is documented in the 
Source Selection Decision Memorandum (SSDM). The SSDM should provide the 
decision, supporting rationale and direction to the contracting officer to make an award. 
(Ref. 3:p. I-19] 

2. Streamlined Source Selection Method 
The streamlined source selection method encompasses the same procedures of the 


formal source selection method and permits the procuring agency the flexibility to award 


contracts based on factors other than price alone. The streamlined source selection method 








is normally reserved for lower cost, less complex procurements. It is also more commonly 


used. In accordance with the DLA Handbook 4105.3, the streamlining can be 
accomplished in four ways; 


1. Streamlining the source selection organization by limiting the number of persons 
involved in the source selection process; 1.e., limiting membership on the SSEB. 


2. Limiting the number of factors and subfactors used in the evaluation process. 
3. Limiting the length of technical proposals to a certain number of pages. 


4. Standardizing the evaluation factors and cniteria to reduce development time and 
amount of explanation and justification required in the evaluation. {Ref. 3:p. II-1] 


All agencies within DoD and NASA have these streamlined procedures which are 
sometimes referred to as the informal source selection method. Minimum dollar amounts 
for use of the streamlined source selection method may vary among agencies. The general 
guidance revolves around the cost of the process compared to the benefit to be gained by 
employment of the process. For instance, an activity may not feel that the support they are 
getting from a previously awarded low price technically acceptable offeror is adequate. 
Prior to recompeting the service, the activity decides through a cost benefit analysis that the 
streamlined source selection method will be used because it will allow the activity to obtain 
better services by using evaluation factors other than price alone. 

The streamlined source selection method is related to initiatives currently being 
pursued at NASA and the Defense Nuclear Agency (DNA) to reduce the Procurement 
Administrative Lead Time (PALT). The Fast Track Procurement program, instituted by 
DNA, strives to reduce PALT to 120 days on service procurements less than $2 million 
{Ref. 13]. It does so by reducing the number of evaluation factors, length of technical 


proposals and incorporating a structured process with recommended milestones. 





As expected, streamlining one aspect of the process will carry over into another. 


For example, reducing the length of the technical proposal reduces the number of 
evaluation board members and the me required to conduct the evaluation. 

The streamlined organization usually consists of the procurement official, or 
contracting officer who acts as the SSA and the SSEB. The SSAC is either eliminated or 
integrated into the SSEB. [Ref. 3:p. III-2] 

G. SUMMARY 

Chapter II provided a definition of the "best value" concept to be utilized in this study 
and discussed when it is appropriate to employ "best value" evaluations. Chapter II also 
provided a definition and discussion of engineering and technical services. The chapter 
was concluded with a generic discussion of acquisition planning and the "best value" 
source selection methods for engineering and technical services. 

Chapter III will provide data from a survey questionnaire used to conduct telephone 
and personal interviews. The survey questionnaire solicited opinions from Government and 
industry personnel on the subject of "best value" and its use in the procurement of 


engineering and technical services. 





Hl. "BEST VALUE" SURVEY QUESTIONNAIRE 


A. INTRODUCTION 

Chapter III is a compendium of responses resulting from telephone and personal 
interviews collected through the use of a survey questionnaire. The questionnaire was 
developed based upon an analysis of the current literature and discussions with 
procurement personnel who utilize "best value" techniques to procure engineering and 
technical services. The questions were left open-ended with the idea of eliciting responses 
which would correlate into meaningful issues for further discussion. Respondents to the 
survey were randomly selected, and include procurement and technical personnel from both 
Government and industry. Prior to the commencement of an interview, the interviewee 
was informed of the nature of the study and advised that comments received by the 
researcher will be kept on a non-attribution basis. Interviewees are listed in Appendix B. 

In addition to the data collected during interviews, information was collected from 
local instructions and training aids, three source selection plans, nine requests for proposal, 
and five source selection decision memorandums representing 17 different contractual 
actions from eight separate activities. The Federal Legal Information Through Electronics 
(FLITE) database was screened for Board of Contract Appeals (BCA) and Comptroller 
General (GAO) decisions for the key terms “best value", "greatest value" and “best buy" 
with regard to service contracts awarded since 1988. 

The data presented are not statistically significant nor do they apply to all DoD 
activities but represent a significant sample of the issues and problems expenenced by the 
interviewees. The format for presentation of the questionnaire commences with the 


question, a discussion of the overall responses, and in several cases, a representative 





sampling of responses. Separate survey questionnaires were utilized for Government and 
contractor personnel. Questions are presented in Appendix C. Major issues that were 


identified during the interviews are defined and analyzed in Chapter IV. 


B. GOVERNMENT QUESTIONNAIRE 
1. Question #1 
How would you define "best (greatest) value”? 
a. Discussion si 
Respondents’ definitions of the term "best value" were consistent with 
published definitions stating that "best value” equates to making the most advantageous 
procurement decision based on factors other than price alone. The majonity defined "best 
value" in terms of a process, alluding to a method of source selection where the 
Government is afforded the opportunity to exercise judgment in considering all factors 
prior to awarding a contract. Adjectives including "subjective," “reasoned,” and "sound 
business" were frequently used as descriptors for the word judgment when defining "best 
value". Outside the Navy, almost 50 percent of the interviewees were not familiar with the 
term "best value" although they were familiar with the process of considering all factors 
including price during proposal evaluation. Four of the interviewees outside the Navy, 
believed that "best value” was a Navy expression used to describe source selection in 
which the Navy used a mathematical formulation combining the scoring of costs and 
technica] factors in making a procurement decision. The Navy was the last DoD agency to 
eliminate the practice of numerically scoring costs. The practice was eliminated several 
years ago. 
Three interviewees offered an interesting caveat that they claimed was a 


common misconception among procurement and non-procurement personnel. "Best value" 











in their opinions, does not mean procuring more than the minimum requirements. Instead, 
it is a method for assessing risk and in turn choosing the offeror with the greatest 
probability for successful contract performance. 
A final caveat presented by a high ranking Navy procurement official considers 
"best valuc” as an objective of the procurement process no matter what the product, 
service, or method of procurement utilized. While the majority of viewpoints tend to 
equate "best value" to offering a premium for a higher quality product or service, this 
viewpoint considers "best value" as an objective. At times, attaining the objective requires 
the payment of a premium. At other times, "best value" may mean competing a 
requirement based on low price or low overall cost. This decision is the essence of "best 
value". 
b. Responses 
Comments reflective of responses received during interviews are listed below: 
"Best value" is a conscious departure from technically acceptable/low offeror 
procurements. It is dependent on risk. How much premium are we (DoD) willing to 
pay to avoid risk. 
"Best value" means getting the most in terms of performance, quality, and 
suitability, not necessarily related to price. From a contract administration point of 


view, it is money put in up front compared to money put in over the long term. 


Under "best value", technical means more than cost. Cost becomes more important 
as technical proposals get closer. 


"Best value" means different things to different people. The Government reserves 
the right to award to other than the lowest bidder based on cost and other factors. 


The "best value” concept involves looking at the entire spectrum of factors in 
making an award. It is based on subjective choices between offeror's proposals. 


To some, "best value" means achieving the most bang for the buck. To others, it 


means making a subjective decision on proposals which the activity feels offers the 
most advantageous procurement decision to the Government. 
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"Best valuc” is not choosing the low bidder. Nor is it choosing the low price 
technically acceptable offeror. "Best valuc" is not a mathematical conclusion. "Best 
value" is a reasoned judgment call based on the reasons of technical ability, 
management, and cost with no excuses and no apologies. 

2. Question #2 
Does your activity use numenical scoring, adjective grades or colors to evaluate 
proposals to achieve "best value?" Why is that rating system uscd? 
a. Discussion 

The three rating systems encountered during this study were: (1) numerical 
scan (2) composite methods in which numerical scores are attached to adjective grades 
or colors, (3) and stnctly adjectival descriptors or colors without any numerical points or 
scales. 

Proponents of utilizing strictly numerical scoring methods feel that rating 
proposals by .1umerical scores is the most direct and simplified method of evaluation. They 
also claim that their customers, the requiring activities, favor numerical scoring as a method 
of differentiating between offerors, i.e., there may be a big difference between a technical 
score of 81 and 86, both of which may compute to an "excellent" adjectival description. 

Composite rating systems employ both numerical scoring methods and 
adjectival or color descriptors. Activities using composite rating systems appreciate the 
ease of point scoring for the technical evaluation committee and the flexibility of adjectival 
or color descriptors offered to the SSA in making award decisions. For example, a 
technical proposal achieving a numerical score between 80-90 points may be assigned a 
grade of "excellent" while a technical proposal achieving a grade between 91-100 will be 
assigned a grade of "outstanding". A composite system is the middle ground between the 
other two systems, employing both point scoring and adjectival or color descriptors. 

Activities utilizing strictly adjectival ratings or colors stress the flexibility 


inherent in a system unconstrained by numbers. Informally, it appears that activities are 
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leaning toward the more flexible rating systems. Two Navy activities encountered during 























this study recently switched from numerical rating systems to strictly adjectival systems. : 
The following is a quote taken from an internal memorandum from an activity currently 

utilizing numerical scoring although they are considering changing to another method. The 

memo is dated 29 January 1991. 

The assignment of numerical weights too often results in the requiring activity using 
it as a crutch rather than biting the bullet and coming up with a strong commitment as to 
whether a higher technical score really offsets a higher cost or in fact, did not offset a 
higher cost. 

Whatever the rating system employed, all activities did stress that the rating 
system is utilized as a tool. The narrative from the technical evaluation committee 
supporting the rating is the key in making the award decision. On several occasions, the 
interviewees inferred that the head of the procurement activity designated the rating system 
as a matter of preference. 

b. Responses 

We use numerical scoring for technical elements only based on a 1000 point scale. 

At one point or another, this activity has used all three (rating systems), although 
we recently stopped using colors since NAVSUP didn't like the way we attached 
meanings to their usage. At present, we are using broad adjectival classes 
supplemented by point scoring. 

We're using points with adjectives. For instance, a score of 85-100 might equate to 
a rating of "outstanding.". The system gives us flexibility through comparison of 
adjectives rather than total point scores. 

We use points exclusively and typically on a 100 point scale. 

Colors, because they graphically represent the overall picture. You do not have to 
go into great depth to understand colors. Colors are more flexible, offering a wider 
latitude to the SSA. Besides, the Air Force has a regulation banning a mathematical 
formula. 


Point system with narrative. 


A color code system is used for a visual aid as a background for which narrative is 
most important. 
























Adjectives. Absolutely no numbers are used during the evaluation process. We 
used numbers in the evaluation process, but switched to adjectives about two years 
ago. At first | was doubtful, but now I would not use any other rating system. 


It is really academic what you use; numbers, colors, or adjectives. It all depends on 
the SSA and what he or she feels comfortable with. 


3. Question #3 
How do you decide a certain requirement requires the use of the "best valuc" 
method? 
a. Discussion 
Responses to this question were consistent with published guidance noted in 
Chapter II. Non-routine service requirements and vague PWSs were normally cited as 
indicators necessitating the employment of "best value" evaluations. Many interviewees 
responded that they utilize "best value" evaluations almost exclusively. Others utilize "best 
value" evaluations less frequently. Several respondents stated that one must review the 
type of contract. Generally speaking, cost type contracts should result from the use of 
"best value" source selection procedures. Augmenting the above indicators on when to 
utilize "best value" evaluations is the input received from requiring activities and contract 
review boards which provide guidance on the type of procurement necessary for a specific 
requirement. 
b. Responses 
It depends on what you are contracting for. The contracting officer usually gets a 
good idea of direction from the branch manager and others during contract review 
panels. The contracting officer is very sensitive to this recommendation. 
This decision requires coordination and a good deal of definition. History has 
shown that engineering and technical services procured on a low offeror basis has led 


to miserable support overall. 


The determination 1s made in conjunction with the technical code requiring the 
support. 


This activity utilizes an acquisition strategy board prior to release of the SOW and 
RFP. The program manager makes the initial call, but it all depends on the nsk 
assessment. 








It depends on the technical approach. With basic services, a responsibility 
determination is adequate. For more complex omnibus or engineering support 
services, "best value" involves looking toward other factors. Contracts and technical 
have had arguments about this. Just because a requirement calls for aerospace 
technology doesn't mean a run of the mill machine shop can't perform the work. 

It depends upon the requirements. Basic maintenance services do not usually 
require “best value". However, more complex services do. Preplanning with the 
technical activity is very important. 

Use of "best value" is dependent on the SOW and complexity of the requirement. 

Depends on how important the technical factors are to the job requirement. 

Consultation with the technical activity requiring the service. You also need to take 
into account the type of service and contract type. With a cost reimbursement contract, 
you will generally lean toward "best value.” 


The determination relates to the service. If service is not clearly measurable in the 
area of acceptability, one will generally lean toward "best value." 


4. Question #4 
What are the factors and subfactors your activity uses for "best value" evaluations 
for engineering and technical services contracts? 
a. Discussion 
Responses to Question #4, during telephone interviews, consistently yielded 
the factors Technical, Management, and Cost as a basis for "best value" evaluations for 
engineering and technical service contracts. An examination of a database of local 
instructions, three source selection plans (SSPs), nine requests for proposal (RFPs) and 
five source selection decision memorandums (SSDMs) from 17 unique contracting actions 
at eight different activities is presented in Table 2 below. The data should be viewed as a 
rough order of magnitude tally only since the researcher arbitranly assigned subfactors with 
reasonably close descriptions to other subfactors. For instance, the subfactors "manpower 
utilization matnx" and "retention policies" were included with the subfactor "management 


plan/approach.” 
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FACTORS AND SUBFACTORS UTILIZED FOR "BEST YALUE" 
PROCUREMENTS OF ENGINEERING AND TECHNICAL SERVICES 






TABLE 2 
FACTORS TIMES LISTED 
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Technical, Management, and Cost are the key factors in "best value” 
procurements for engineering and technical services although, in this sample, Management 
is generally listed as a subfactor of Technical. Other key subfactors are: 1) Corporate 
Experience/Background, which can be viewed as the level of experience in identical or 
similar contracts (these data are used by several activities to perform a qualitative past 
performan ~- assessment on the offeror); 2) Technical Approach/ Technical Understanding, 
which equates to an understanding of PWS requirements; and 3) Personnel Qualifications 
of key personnel represented by resumés supporting minimum requiremerts for experience 
and education of proposed key personnel. The subfactor Corporate Resources/Facilities 
evaluates the offeror's facilities and equipment, and Sample Tasks allow the offeror to 
propose solutions to tasks which have been previously assigned or may be assigned in the 
future. In accordance with the FAR, Cost/Price is a required evaluation factor for all 


proposals [Ref. 8:p. 15-19]. 
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§. Question #5 
How are past performance data collected and utilized in "best value" source 
selection for engineering and technical services contracts? 
a. Discussion 
According to interviewees, past performance is a factor or subfactor in the 
overwhelming majority of "best value" evaluations for engineering and technical services 
contracts. The process of collecting past performance data commences with the 
instructions to offerors, Section L, of the RFP. Section L requires offerors to submit a 
detailed listing of previous contracts or experiences similar to the current PWS. This 
requirement is normally listed under the evaluation factor "Corporate Experience/ 
Background". Once the data on experience are collected, the activity utilizes these data to 
research an offeror's past performance. Past performance data are collected through the 
use of questionnaires or telephone queries to the procuring activity or the contract 
administration office having cognizance over previous efforts. Past performance data are 
typically related to the contractor's adherence to proposed cost, schedule and general 
cooperativeness. An example of a questionnaire utilized by NAVAIR to collect past 
performance data is presented in Appendix D. The process of past performance data 
collection can be either formal, under a structured program, or informal where one: -tivity 
contacts another to check the validity of data provided by the offeror and for general 
information on a contractor's performance. Both methods were encountered during this 
research. 
Unlike past performance data for tangible items, there is currently no rating 


system like Red/Yellow/Green, Blue Ribbon Vendor, or the Contractor Performance 
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Rating System for engineering and technical services contractors. This fact is related to the 
inherent difficulty of assessing quality and delivery schedule compliances of service 
contractors performing under vague PWSs. 

The Naval Sea Systems Command (NAVSEA), Washington, D.C. recently 
implemented a pilot program, the Contractor Performance Index (CPI), to fulfill the need of 
providing assessments on the past performance of engineering and technical services 
contractors. The program utilizes supportable and verifiable data from Contracting 
Officer's Technical Representatives (COTR) to assess the positive and negative aspects of a 
contractor's past performance on a given contract over a specified period of time. CPI 
allows for, "Subjective assessments concerning the causes or ramifications of the 
contractor's performance (which) should be provided; however, speculation and conjecture 
should not be included." [Ref. 14:pp. 1-2] 

One of the major issues noted during interviews germane to the use of past 
performance data was the concern of past performance data being substituted for a 
responsibility determination. In fact, over 50 percent of the interviewees presented with 
Question #5 said that care must be taken not to utilize past performance data in place of a 
responsibility determination. What this means, is that if a contractor is found responsible 
to provide the services desired and has a certificate of competency from the Small Business 
Administration, the Government cannot subsequently utilize data based solely on past 
performance to override the responsibility determination during the source selection 
evaluation process. Two interviewees responded that they believe responsible small 
businesses with certificates of competency indicating responsibility were eliminated under 


the guises of "best value" source selections based on past performance. This was 
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evidenced in a Comptroller General decision in the matter of FLIGHT International Inc., 
discussed in Chapter I] and Comptroller General decision B-231607 of 20 September 1988 
in the matter of Sanford and Son [Ref. 3:p. II-13). 

All activities encountered during research attribute only a small portion of the 
evaluation's total weight to past performance. In reference to the sample of contracting 
actions collected during this research and discussed in Question #4, it was found that the 
weightings of the subfactor Corporate Experience/Background ranged between 2.5 and 15 
percent of the total evaluation with the majority of the weightings ranging between 5 and 10 
percent of the total evaluation. As a result of this relatively small weighting, the past 
performance subfactor should not overwhelm other factors during evaluation. 

A recent trend in past performance evaluations discovered during personal 
interviews, is the recognition that all past performance data provided will not be 
complimentary toward the offeror. In other words, the Government is urging contractors 
to present all past performance data good or bad. In cases where poor past performance 
data exist, offerors are encouraged to provide data advising the procuring activity how the 
problems were addressed, corrected and the current status. If the corrective action has 
merit, the offeror will not be downgraded for that particular discrepancy. An example of 
this trend was offered by a senior Navy acquisition executive who identified two offerors, 
both of whom were delinquent on their last contract. The first offeror maintained the 
reason for delinquency was late Government Furnished Material (GFM). The second 
offeror also maintained the reason for delinquency was late GFM, however the second 
offeror went further by outlining the actions taken by the contractor in an attempt to work 
around the problem. The anecdote was ended with a question "Based solely on the 


example, which offeror would you select?” 
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b. Responses 


This a delicate area. In either case, whenever past performance is good or bad, it 
wil] be given 110 percent of the evaluation by the technical evaluators. You must be 
careful not to assign an inordinate amount of weight to past performance since it could 
amount to a responsibility determination. The key is to ensure contractors understand 
past performance is important and tell them so in the solicitation. It needs to be one of 
the foremost things in their minds while performing a contract. 


We call for past performance data in the RFP. Contractors submit 36-60 months of 
prior performance data. We evaluate this performance data along with the same 
information we get from the contract administration office. It can equate to a damming 
- (poor) mark if we find out poor past performance by ourselves and it was not presented 
by the contractor. A contractor can fix the stigma of poor past performance by 
presenting a management plan in enough detail that tells us how they intend to correct 
the deficiencies. 


Past performance is a management factor. Call past references. If you only get one 
bad reference or no references, then subtract points for past performance. 


That is a tough question. Corporate experience with regards to products is much 
easier. It is more difficult for services because we cannot normally say something like, 
"Well 85 percent of your products were on time without deficiencies" for service 
contracts. How to evaluate past performance must be written into the SSP. For this 
activity, past performance is normally written in as a major category. 


You need to be careful you do not cross the line into a responsibility issue. For 
instance, not asking for information on how the offeror would complete the effort 
under this solicitation and concentrating instead on past performance on other contracts. 


Past performance is a touchy area since you can get into responsibility issues. It 
could also give you false readings. For instance, there may be a different management 
team for this contract than for one that was completed previously. 


Services are not yet incorporated in the contractor performance assessment rating 
system. We use questionnaires to collect data. 


The plan on how to evaluate past performance must be tailored to each buy. We 
look at how responsive headquarters was to local management problems. Past 
performance also needs to be reviewed in light of what type of contract was let in the 
past. For instance, you need to evaluate past performance differently under a fixed 
price effort compared to cost reimbursement. 

6. Question #6 
What are some of the problems or issues your activity faces with "best value" 


evaluations for engineering and technical services contracts? 


39 


a. Discussion 

The personnel interviewed were unconstrained by the researcher and 
encouraged to provide as much information as they desired. Many interviewees provided 
three or four problems, while others provided only one. Several interviewees expounded 
by providing a description of a problem and an example while others provided only one 
line descriptions. The severity of the problem in relation to the overall process was not 
assessed. 

Overall, responses to this question were varied but generally focused in the 
areas of acquisition planning and proposal evaluation. Although rarely mentioned 
specifically, a pervasive problem perceived by the researcher was a lack of time and/or 
resources to properly plan and conduct a "best value" evaluation. 

The list of problems is segregated into three categories; 1) Acquisition 
Planning, 2) Proposal Evaluation, and 3) Other. A brief description of the problem and the 
number of times it was mentioned is listed below each category. Problems mentioned most 


often are further defined and analyzed in Chapter IV. 


CATEGORY 
Acquisition Planning Times Noted 
1. Requiring activity feels constrained by a 5 


technical/cost tradeoff during "best value” evaluations. 
Instead, the evaluation should focus entirely on the 


technical proposal. 

2. Improper planning of evaluation criteria used in 10 
proposal evaluation to differentiate between offerors. 

3. PWSs written restrictively allowing unfair advantage 4 
to incumbent. 

4. “Best value" method employed when low price/technically 5 


acceptable offeror strategy would yield the same result. 


5. Too many subfactors add complexity to process and 3 
dilute results. 


Poor communications between procuring and requiring 
activities in acquisition planning phase. 


Evaluation criteria established which do not deal with 
true quality considerations between offerors. 


Proposal Evaluation 


Contractor gaming on cost and technical proposals. 


. The technical evaluation committee is not trained nor 


is it sensitive to the proposal evaluation process. 


SSP formulated by different group other than personnel 
who actually conduct the technical evaluation. 


Problems with past performance data use in evaluations; 
1) using past performance in evaluations as a guise for 
responsibility determinations, 2) fairness issues associated 
with using past performance to downgrade a technically 


acceptable offeror with no experience, and 3) evaluating past 


performance on different management teams within 
the same corporation. 


Contractor bait and switch tactics with key personnel. 


Resumé requirements written and evaluated for specified 
personnel, 1.e., personal services. 


Cost realism; only a paucity of guidance available on 
how to complete. 


Other Issues 


Contractors distrust the "best value" source selection 
method. "Best value" represents a method the 
Government uses to select the contractor 

they wanted in the first place. 


Government sending the incorrect signals to contractors 
by cutting budgets, yet continuing to offer premiums. 


Legal considerations. "Best value" evaluations offer too 
many areas to protest. 


. Centralization of contracting activities, NRCCs, is not 
an Organization conducive to obtaining the "best valuc”. 


Contract administration: The Government is not getting 
what it is paying for in higher rated technical proposals. 
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6. Too many changes to the procurement system. Policymakers 5 
attempting to transform an inherently subjective 
process into an objective one. 


i) 


7. Funding: Requinng activities limit requirements based on 
funding constraints. When more funds become available 
requiring activities desire to increase scope of contract. 

7. Question #7 
How does your activity determine the need to pay a premium for a technically 

superior proposal over a proposal which is technically acceptable, yet lower priced? 

a. Discussion 

The essence of the "best value" concept lies in the reasoned judgment of the 
SSA in assessing relative strengths and weaknesses of competing proposals. In 
determining whether to pay a premium for a higher rated technical proposal, the 
interviewees resoundingly emphasized the use of this judgment. Of course, in order to 
assist in making the judgment more educated, many interviewees stressed the importance of 
a cost benefit tradeoff analysis which assesses the impact to the Government resulting from 
a technically superior proposal with the highest score or rating, in comparison to the 
proposal which is technically inferior but provides the lower evaluated cost. This tradeoff 
analysis is performed by the technical evaluation committee who assesses and documents 
the technical differences in a quantitative format. Essentially, they determine how much 
more the technical superiority is worth in dollars? 

A more objective measure of quantifying the cost/technical tradeoff 
encountered during this study, was in the form of a linear relationship where the acquisition 
planning process established a minimum acceptable technical score. Additional technical 
points over the minimum acceptable score are equated with a percentage premium the 
activity would be willing to pay. For instance, if the minimum acceptable technical score 


was 80 on a 100 point scale and technical was two times more important than cost, the 





activity would be willing to pay up to a 40 percent premium over the lowest acceptable 
technical proposal, or 2% cost for each point in technical above the low acceptable offeror. 
This premium is advertised in the solicitation. It should be emphasized that even with this 
objective measure, personnel at the activity stressed that the use of judgment continues to 
be the key to the tradeoff process. 

One final area where reasoned judgment plays a key role is when technical 
scores are very close. If this occurs, a determination may be made to call the technical 
proposals equal and the contract award is made on the basis of lowest evaluated cost. 

b. Responses 


This is not really a big problem. We use a premium method which represents a 
linear relationship between cost and technical. For instance, one point of technical is 
worth two percent of a cost premium. If score is within one or two points, we'll 
usually call the proposal equal on technical merit and award to the low offeror. 


We try to use the best engineering and technica] judgment. As an SSA, I require 
the chairman of the SSEB to explain differences to me in terms I can understand. If I 
can understand the differences, then other people can too. I call it the reasonableness 
test. 


You need to look at the discriminators which were set up in the SSP and 
communicated to offerors via the solicitation. What are the differences in? There is no 
formula for it. It all depends on deliberative and reasoned judgment. 


Mathematical formulations to arrive to this point are only tools. We are not 
compelled to award to an offeror based on high score alone. The contracting officer 
and chairman of the SSEB present all the facts to the SSA who utilizes educated 
judgment. 


This happens frequently. You need to go back to the SSEB and have them tell you 
how much the extra technical is worth in dollars. They (technical evaluators) do not get 
the difference in dollars up front. They could possibly manipulate the system if they 
had the costs. 


We require a cost impact statement from technical which basically equates to a 
cost/price tradeoff analysis. You can give them costs before hand or ask them to justify 
an amount the extra technical is worth. If technical is very close, based on narrative, 

- we'll go with the low cost offeror. 
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There is no specific guidance on the mechanics of the tradeoff analysis since we do 
not want to tie the hands of the SSA. I have seen it done both ways. Supplying 
technical with the costs at this point and not supplying them with costs. Whatever the 
method, the narrative is critical. 

The chairman of the SSEB has to be able to communicate that for this much more in 
price these are the advantages we will obtain so that a Colonel or General (SSA) will 
understand. 


It all boils down to a judgment call by the SSA which is not as objective as some 
might be led to believe. 


8. Question #8 
How does your activity conduct the cost realism analysis for engineering and 
technical services contracts? 
a. Discussion 

Responses to this question emphasized the use of independent Government 
cost estimates based on historic and similar efforts. Should cost and most probable cost 
estimates were also mentioned utilizing DCAA provided labor, overhead and general and 
administrative rates, the Department of Labor's wage determinations and Government civil 
service wage scales as standards. Overall, responses typically described the cost realism 
analysis as a time consuming process many times exacerbated by company ventures 
requiring the use of subcontractors whose costs are also subject to analysis. 

Examples of cost realism analyses reviewed at several activities revealed that 
they concentrated on the major cost elements; direct labor, overhead, and general and 
administrative expenses. Material, other direct charges and escalation rates are usually 
"normalized" for engineering and technical services contracts meaning that there should be 
no logical differences between contractors on these costs. As a result, all offerors are 
evaluated on the same normalized cost figures. 

Direct labor rates were usually compared to DCAA audited labor rates and rates 
from previous or similar contracts. For the Service Contract Act covered labor categories, 


there is normally a comparison of proposal rates with those published by the Department of 





Labor. Of course, comparisons of rates are not always straight forward. At times, labor 
categories provided by the Government do not match the nomenclature of categories 
provided by the contractor. Section L of the RFP normally requires a cross reference to 
match labor categories represented in the RFP to those proposed. 

Overhead rates were compared to DCAA audited overhead rates and overhead 
rates on previous contracts. Several interviewees noted that there was a "twist" to 
comparing overhead rates in this type of analysis where the objective is to ensure that the 
proposal is high enough to cover the cost of the contract performance. This is in contrast to 
DCAA's normal concern which is to ensure that the Government is not being overcharged. 
Fringe benefits is another key concern when evaluating overhead rates. The concern is 
with the ability of a contractor to retain quality employees without providing adequate 
fringe benefits such as vacations and medical plans. DCAA audits and historical costs are 
also consulted with regards to uncompensated overtime, which can be used as an indicator 
of the expected quality of work and retainability of emplovees. 
| Opinions from interviewees and a review of a small sample of actual cost 
realism analyses indicated that proposal costs are adjusted upward on a large percentage of 
proposals. In fact, one interviewee estimated that costs were adjusted upward at least 75 
percent of the time based on a cost realism analysis. This estimate may be on the upper end 
of the spectrum, but as evidenced from the number of respondents noting contractor 
gaming on costs as a problem, the percentage of upward adjustments of contractor costs is 
relatively large. 

b. Responses 


Labor rates are the important things in this type of contracting. You really need to 
look at and compare expertise and quality with the rates proposed. 


45 


Contractor gaming makes cost realism difficult. Companies form joint ventures 
with others where nobody with resumés provided is assigned to the company being 
evaluated. Companies set up new cost centers. There is really no one way to do cost 
realism and answer all the questions. . 


This activity uses SCA dollar wage advisories and historic rates. Basically a 
thorough should cost analysis. 5 


It is difficult to find material on cost realism. Each person does it different. For 
this contract, I developed my own approach establishing a matrix including past 
contract information and rates from a similar contractor in the area. 

Cost realism requires balancing the cost proposal and the technical proposal to 
ensure things match up. Normally it does not require the assistance of the technical 
people to do this. 

We get very involved in the personnel costs proposed to ensure these costs match 
the technical proposal. There have been instances where the entire proposal has been 
refuted on a mismatch between the two. 


Many contractors are reliable, others are not. Lots of extra work, time, and effort 
are expended on cost realism because you cannot rely on their (contractors') data. 


Since we do not always have an independent Government cost estimate, we rely on 
a most probable cost scenario where the customer provides an estimated labor rate, 
labor estimate, and materials. : 


Our job is to figure out if costs proposed are realistic, fair, and reasonable, and 
there are too many intricacies to making up specific guidelines on the subject. . 


Cost realism analysis require saavy and experiencc. 
Cost realism requires ethical behavior from both the contractor and the 
Government. 
C. CONTRACTOR QUESTIONNAIRE 
1. Question #1 
How would you define "best (greatest) value"? 
a. Discussion 
Definitions provided by contractors were completely consistent with definitions 
provided by Government personne]. A total of seven contractors were interviewed for this 
study, and all were familiar with the term and the concept. All provided services to the 


Navy. 





2. Question #2 
How do you feel about the "best value" concept utilized by the Government in 
awarding contracts? 
a. Discussion 

Consensus among the sample of contractors contacted for this study believed 
that the "best value" concept is a good one for two reasons. First, its premise is based on 
high quality over low price, and second, because it is a method of protecting responsible 
offerors from offerors who bid unrealistically low rates when competition is based on price 
alone, i.e., a $7 per hour engineer. A program manager for a major support contractor that 
was recently awarded a "best value" contract, admitted that he knew his firm was not the 
low bidder after the award. He also admitted that due to the "best value” award, he feels 
the firm is more motivated to perform quality work rather than focusing on lowering costs. 

Unfortunately, contractors do no always believe that the Government is 
focusing on quality in lieu of costs. This was mentioned by both Government and 
contractor personnel. Instead, contractors accuse Government procurement officials of 
being overly concerned with the low bid. Some of this perception is true. One 
Government interviewee admitted how he was unable to eliminate a technically 
unacceptable offeror from the competitive range because "the price was too good." 

The perception held by contractors of the Government being concemed solely 
with price, leads to cost and technical gaming sometimes attributed to "remaining 
competitive" and "fighting fire with fire." As one contractor stated, "Gaming is a result of 
the rules of the road and contractor's mistrust of the process. I feel Government should 


award on the basis of technical superiority only!" 





b. Responses 


I do not believe Government is always buying the "best valuc". Everyone needs to 
change their mindset before "best value" will happen. The current mindset is on low 
cost. 


I think it is great! There have been a great proliferation of companies bidding jobs 
they cannot do. "Best value" is a way to correct it. 


"Best value" is easier to do with something tangible like widgets. It is more 
difficult in a softer area like services. With widgets you van look at MTTF, warranty 
and other things. 

"Best value" is the only way to purchase something that is not under detailed 
specifications. Unfortunately, pressure is on the Government contracting officer to 
award to the low offeror. The Government can do a better job in this area. Too many 
contracts are awarded to the low bidder. 

The concept is right on because picking contractors based on the low bid is no 
good. Under low bid you get the $7.00 per hour engineer and firms have to fight fire 
with fire. I sometimes think "best value" 1s only lip service though. 

"Best value" is terrific if the Government can stick to it. 

3. Question #3 

Does your firm have any problems with "best value" solicitations let by DoD for 
engineering and technical services? If so, what are they? 

a. Discussion 

The responses to this question presented an almost even split of contractors 

who believe "best value" solicitations are clear and those who do not. Half of the 
respondents believed that solicitations offered ample information regarding the evaluation 
criteria and what was important to the Government. The other half do not believe that the 
evaluation criteria are clear partly due to, as one contractor stated, "consistently changing 
language in Sections L and M of the RFP.” Although procuring activities have attempted to 


standardize evaluation factors and "best value” wording in RFPs, there is little 


standardization between different activities. Compounding the confusion are the different 





rating systems used at different activities. Contractors, constrained by resources and faced 
with what some term “cut throat" competition, need to determine where to place their 
emphasis when constructing a proposal. 

b. Responses 


Government tries to define pre-established rates and levels of qualifications. Many 
times they (Government) fail on both counts. 


Past performance if evaluated is difficult for the new guy on the block. You may 
_ have a new guy who is great . You penalize him because of little past performance data. 


I am never really clear what the evaluation criteria are. NASA does a good job in 
this area, however, DoD is somewhat haphazard. Good criteria means it is easier to 
understand what the Government will evaluate. 

I have no problems with the solicitations. What I do have problems with is that I 
lack confidence that the Government is actually going to award "best value". There are 


probjems in the environment. Rules continue to change. It used to be low offeror with 
discussions, now it is "best value" without discussions. Sometimes it is hard to keep 


up. 
4. Question #4 
How do you feel about awarding contracts on a “best value" basis without 
discussions? 
a. Discussion 
Contractors and Government personnel favor awarding contracts without 
discussions because it can save resources by streamlining the process. However, in 
responding to this question, all contractors cautioned that awarding without discussions is 
good policy only when PWSs, instructions to the offeror, and evaluation criteria are well 
defined. These criteria are typically not present for engineering and technical services 
unless the effort is mature and well established. This researcher encountered only one 
engineering and technical service contract awarded without discussions. It was awarded to 
an incumbent who was evaluated with the high technical score and low cost. Furthermore, 


in this situation, it was believed that no other competitor was susceptible to award of the 
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contract. However, a supervisory contract specialist at one activity estimated that up to 
25% of the contracts awarded at his activity are awarded without discussion to other than 
the low priced offeror. 
§. Question #§ 
In your view, does the Government do an adequate job of cost realism analysis? 
a. Discussion 
Two contractors cited the Government's poor history in performing a cost 
realism analysis through the example of the $7 per hour engineer. However, at present, the 
overwhelming majority believe that the Government is performing more accurate 
assessments of contractors' actual costs. Moreover, contractors now believe that the 
Government is comparing cost and technical proposals for cost realism purposes. 
Of course there are still a number of contractors who doubt the effectiveness of 
a cost realism analysis. One contractor responded that he sees "a lot of high sounding 
words about cost realism in RFPs, but I] sometimes believe it is only lip service," citing an 
example where his firm was the low price offeror prior to the Government adjusting their 
proposal price upward. He felt that his firm lost the contract largely because of this price 
adjustment and stated that after such an occurrence "it always makes you wonder if the 
Government is only sustifying a way to pick the contractor it wanted in the first place." 
Another contractor responded that "industry knows more about its costs than does the 
Government," raising the issue of industry's opportunity to justify its cost prior to award 
or request for a BAFO. 
Finally, every contractor interviewed brought up the topic of uncompensated 


overtime (UCOT) while discussing the cost realism analysis. All contractors in this small 





sample believe that UCOT usage in proposals should be prohibited by regulation since it 
allegedly “destabilizes the playing field". The UCOT issue is further defined and addressed 
in Chapter IV. 
b. Responses 
(Cost realism) Used to be horrendous. Resumes of the best people were in the 
technical proposal but not in the cost proposal. UCOT problems- how much UCOT is 
actually being applied to the contract in question? Before the technical and cost people 
never got together. Today, I think that is different. People are starting to talk to one 
another in my estimation. 


UCOT is still a problem judging by input from the field, but it is now popping up in 
different places like Huntsville, Alabama. 


Sometimes the contractor has a better understanding of what the effort will cost than 
does the Government. We have more years of experience. 


On our current contract yes. The Government should watch UCOT though because 
in my estimation it leads to a lower quality of work. 


Government does a good job if they want to. Problems in evaluating cost realism 
are in the area of UCOT which is voodoo. Even those who bid it do not understand it. 
UCOT is screwing over the Government and the contractors. It is hard to justify and 
cost. It forces Government to look at the bottom line. 

D. SUMMARY 

Chapter II] discussed and presented responses to a survey questionnaire used to solicit 
responses from Government and industry personnel. Data collected from a sample of 
SSPs, RFPs and SSDMs were also utilized to respond to several questions in Chapter III. 


Major problems and issues discovered as a result of the survey questionnaire are the subject 


of Chapter IV. 
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IV. PROBLEM DEFINITION AND ANALYSIS 


A. INTRODUCTION 

This chapter will focus on the definition and analysis of the most frequently mentioned 
problems associated with "best value" evaluations identified in Chapter III. The most 
frequently mentioned problems discussed in this chapter are: 1) the selection and weighting 
of evaluation criteria, 2) the technical evaluation process, 3) contractor gaming in cost and 


technical proposals and 4) contractor distrust of the "best value" source selection process. 


B. “ROBLEM #1: EVALUATION CRITERIA 

The determination and weighting of appropriate evaluation criteria which differentiate 
between offerors for use in "best value” evaluations. 

1. Problem #1 Defined 

The determination of the proper evaluation criteria for use in source selection is a 

critical aspect of the acquisition planning process. It was mentioned as a problem by 
several respondents to the research questionnaire as well as by NAVSUP, which noted 
Problem #1 as an area of concern from both the technical and procurement personnel. This 
was documented in a report by the Naval Supply Systems Command Greatest (Best) Value 
Source Selection Process Action Team which stated a "Need for the joint development of 
source selection plans which lead to differentiauon between offerors and emphasis on past 
performance and quality" (Ref. 15:p. 4]. Problems in this area were also observed during 


visits to contracting activities. 





2. The Importance of Evaluation Criteria 

Evaluation cnteria are essential in providing a framework necessary to ensure that a 
quality service is obtained. The SSP is used to communicate these criteria plainly to source 
selection officials and is the crucial document in preparing the RFP which communicates 
these criteria to potential offerors. 

It is important that the evaluation criteria be individually tailored to each 
procurement in accordance with FAR 15.605. The evaluation criteria should also be 
limited so as to select criteria which will focus on the important issues for the successful 
accomplishment of the PWS. Proper selection of evaluation criteria is necessary to 
effectively discriminate among the proposals and permit the correct ranking of the 
proposals in the order of their value to the Government. 

Properly identifying the evaluation criteria increases the probability of evaluating 
and selecting the most advantageous contractor. If the requiring activity knows what 
aspects an offeror's proposal should address and how these aspects will be evaluated prior 
to the technical evaluation, then chances are much greater of conducting a disciplined 
source selection and making the most advantageous procurement decision. If the 
evaluation criteria are poorly identified, there may exist inconsistencies in the technical 
evaluation process which increases the time required to perform the evaluation and the 
possibility of protest by unsuccessful offerors. 


3. Deficiencies Mentioned by Contracting Personnel Regarding the 
Evaluation Criteria in "Best Value" Procurements 


The following comments collected during this research reflect the problems with 
evaluation criteria for use in "best value” procurements: 


We utilize 14 standard factors and subfactors to evaluate offerors. Unfortunately, 
this sometimes has an averaging effect. 
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It is sometimes hard to discriminate between offerors of engineering and technical 
services. Sample tasks are a good way to do so if they are reasonable and evaluation of 
them is critical. 

Evaluation factors (identified by the requinng activity) are not always intrinsic to the 
job. For example, does it really matter if the contractor's facility is close by or far 
away? Sometimes it makes no difference. 


Using the right evaluation criteria is "motherhood" for all procurements. They 
(evaluation criteria) are usually harder to wnite for "best value" procurements though. 


One of our biggest concerns is to ensure we have finite discriminators and actual 
tasks which are needed to be performed. For instance, in the area of resumes for key 
personnel, we have to watch out since we are buying a level of expertise and not 
personal services. 


The hard part about "best value" is setting up evaluation criteria which will get you 
there. 


Several times, (evaluation) criteria made up by the requiring activity do not make 
common sense. Why do you need someone with that much experience? 


One has to be careful not to lose the forest through the trees when identifying 
evaluation criteria. It is really important for technical to understand their requirement 
and what they consider important to accomplish it. Many times technical chooses too 
many criteria. 


4. Examples of Problem Situations with Evaluation Criteria 
Encountered During this Research 


Examples of problems with "best value" evaluation criteria encountered during 
research are presented below. Each example demonstrates how the selection of evaluation 
criteria can affect the procurement of the most advantageous procurement decision. The 
first example presents the standard cnitena and subcniteria used in evaluations at one activity 
which at times has the effect of diluting evaluation results. Diluting results means it is more 
difficult to discriminate between offerors. The second example demonstrates how too 
many evaluation criteria, poorly presented can complicate the technical evaluation process 
and make it more difficult to determine the most advantageous procurement decision. The 
final example, presents technical criteria discussed at a meeting attended by the researcher 


between a requiring activity and contracting personnel regarding evaluation criteria for a 


"best value" procurement. Asa result of the meeting, the acquisition strategy changed from 
"best value" to a low cost technically acceptable. 
a. Example #1 
The following are technical evaluation critena and subcriteria, quoted from a 
standard source selection plan used at one activity. These criteria sometimes have the 
potential effect of diluting the results of the technical evaluation. The sample plan states 
that the factors used in the technical evaluation "shal] include all the evaluation factors 
described below:" 
1. CRITERIA 
a. Technical/ Management 
(1) Technical Approach (Criteria) 
Subcriteria: 
(a) Understanding and Approach 
(b) Specific Technical Capability 
(c) Problem Areas 
(d) Unique Ideas and Capabilities 
(e) Workforce Loading Plan 
: (f) Start-Up Pian 
(g) Sample Tasks 
(2) Experience (Critena) 
Subcritena: 
(a) Personnel Qualifications 
(b) Corporate Experience 
(3) Management Approach (Cnitena) 
Subcnitenia: 
(a) Contract Management 
(include subcontractors, if any) 
(b) Interface 
(c) Organization 
(include subcontractors, if any) 
(d) Cost Management 


(4) Facilities and Resources (Criteria) 
No subcntena 
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The problem with employing the above number of criteria and subcriteria to 
evaluate proposals is the potential averaging effect. In total, there are four criteria and 13 
subcriteria for utilization during a technical evaluation. Attaching weights to each of the 
above criteria and subcriteria dilutes the effect of what may really be important to the 
requiring activity and therefore makes it more difficult to discriminate between offerors. In 
this situation, requiring activities should be afforded the opportunity to choose only the 
most important criteria for the evaluation. All criteria and subcriteria may not require 


review in each evaluation. 
b. Example #2 


This example demonstrates how too many criteria, poorly presented, can 
complicate the technical evaluation and increase the time required for the source selection 
process. The result, limits the effectiveness of a “best value" evaluation since technical 
evaluators may not be certain as to which considerations are the most important to the 
requiring activity. In this example, there are far too many cniteria or "considerations" to 
review in completing the technical evaluation. In addition, there are no weights, 
quantitative or qualitative attached to any of the considerations. As a result, it is not 
possible to determine which of the considerations are important and which ones are not. 
One evaluator may grade the factor very high based on one or several criteria, while another 
evaluator may grade the proposal low based upon other criteria. The following is quoted 


directly from the technical evaluation plan. 





PLAN TO ACCOMPLISH WORK 

Maximum Score Available- 10.0 points 

Each evaluator shall review and score the proposal of the offeror based on the 
following subfactors. 

Subfactor 1: Organization 

Maximum Score Available- 5.0 points 

Structure of the organization, or proposed organization, and relationships to the 
task of the parent company, and subcontractors (if any) and interface. 


Considerations: 

Does the proposal detail and specify clearly that the offeror has a thorough 
understanding of the requirement? Are the technical problems understood? Are valid and 
practical solutions set forth? Is there an organizational structure chart in the proposal? Do 
the lines of communication appear clear? Are relationships to tasks of the parent company 
fully explained? Does the organizational structure contain any ambiguities when compared 
to the other sections? Are lines of communication clear between key personnel and 
management? How will the contractor communicate with the requiring activity? Are the 
lines of communication clear? How will the contractor communicate with any proposed 
subcontractors? When compared with resumes, are key personnel authorized appropriate 
responsibility? Does the proposal indicate that the appropriate number of personnel are 
employed on a permanent basis and available during key workload periods? Is delivery 
order processing discussed? How will the order be processed by the offeror and 
disseminated to key personnel? Are any contractual arrangements other than cost 
reimbursement contemplated for this proposal? Are there any firm fixed price or time and 
materials type subcontracts proposed? If proposed, is there a rationale to document any 
proposed technical advantage? Does the proposal indicate a start up plan to fully perform 
within sixty days after award of the contract? Is the plan realistic and detailed? Does the 
proposal explain the proper methodology and procedures to responding to delivery orders? 
Do these plans show management responsibility? Are uncompensated overtime hours 
proposed? Is the appropnate information provided as indicated in section L? Are the 
uncompensated hours proposed a substantial benefit to the Government? How do 
subcontractors interact with the pnme? Is this documented sufficiently? Attachment (2) 
sets forth a checklist to assist the evaluator in determining the evaluated score. 


Evaluator's Score 
Evaluator's narrative: 


The above example includes 27 considerations for each proposal. Some of the 
considerations are subjective, i.e., Do the lines of communication appear clear? Other 
considerations are more objective, i.e., Is an organizational chart presented in the proposal? 
Based on the number of considerations and the mix of subjective and objective measures, it 


may be extremely difficult if not impossible to differentiate between offerors. Which 








offeror presents the most advantageous procurement decision depends upon which 


considerations the technical evaluator determines to be the most important. 


In addition to subfactor #1 presented above, "Organization", the total technical 


evaluation consisted of nine subfactors with approximately 20 considerations each. 


Besides complicating the evaluation process, the amount of time and resources required to 


assess and document each proposal evaluation may be excessive. 


A less complicated format for presenting and rating evaluation criteria for a 


technical evaluation was extracted from another sample evaluation plan and is presented 


below. This example is for the subfactor "technical approach- sample tasks". 


SUBFACTOR: TECHNICAL APPROACH 
OFFEROR 

EVALUATOR 

PART 2- SAMPLE TASKS 

SAMPLE TASK # 


To what extent did the proposal demonstrate understanding of the feasibility and 
ability to successfully perform the task in the following responses: 


6. 


7 


Rating* 
O B A M U 


. The proposal provides a description of the 


possible areas to be investigated in researching 
the task. 


. The proposal provides a detailed descniption 


of the technical approach including a detailed 
step-by-step procedure and methodology 
which would be used in accomplishing the task. 


. The proposal identifies additional information 


that would be required to perform the task. 

The proposal contains a detailed work plan for 
implementation. 

The proposal contains a product outline describing 
what would be the expected deliverables and results 
from this task. 

The proposal contains manhours by labor category. 
Overall rating. 


*Ratings- O= Outstanding, B= Better, A= Average, M= Marginal, U= Unsatisfactory 
Narrate any strengths and weaknesses below: 








The simpler format presented above allows the TEC to reduce the time required 


in a technical evaluation. It also clarifies for the evaluator what the requiring activity views 
as important in an understandable format. The smaller number of considerations also 
reduces the probability of subfactor dilution. Note that all considerations receive an 
adjectival rating with an overall rating included at the bottom of the evaluation sheet. Each 
consideration is not weighted but reviewed in lieu of all other considerations. In the prior 
example there would have been 27 ratings, one for each consideration. This simpler format 
permits the evaluator to use judgment during the evaluation by only requiring the evaluator 
to review six considerations before making a determination on the overall rating. 
: c. Example # 3 

The final example was observed during a meeting between a requiring activity 
and contracting personnel when discussing the use of "best value" to procure a series of 
environmental tests. The following technical evaluation criteria were submitted to the 
contracting activity for approval prior to the solicitation. 


TECHNICAL EVALUATION CRITERIA 


Factor Weightings 
1. Technical 75 


~) 


MAMAOD NWNMANNO AAA NAAN 


a. Environmental and safety tests per SOW 
1) 28 day temp and humidity 
2) transportation vibration 
3) fast cookoff 
4) slow cookoff 
5) bullet impact 


b. Qualified personnel per SOW 
1) senior test engineer 
2) test engineers 
3) test technicians 
4) security staff 


t\ 


c. Adequacy of facilities 
1) proper test equipment 
2) adequate security and storage 
3) handle test items safety 


mb 
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d. Performance of similar tasks 

1) depth of understanding 

2) realistic techniques 

3) technical approach 

4) experience 

5) data requirements 

The problem with the above cniteria is the fact that they are based more on 
responsibility type issues than on issues related to varying degrees of quality or technical 
merit. In essence, the criteria focused more on whether an offeror, based on his/her 
understanding of the tasks involved and the facilities required could perform the tests 
required in the SOW. Most of the criteria could be addressed with a "yes" or "no" 
response, i.e., either the offeror has the ability to fulfill the SOW or he does not. The 
meeting determined that there was little differentiation based on the accuracy, timeliness or 
quality of the tests required by the SOW. 

After approximately one hour of honest and upfront dialogue by both parties, it 
was decided that a low cost technically acceptable approach for the acquisition was optimal. 
It is this researcher's opinion that a key in the decision was the contract specialist's 
understanding of the technical requirements. Without this understanding and the ability to 
ask relevant questions of the requiring activity, a "best value” strategy may have been 
pursued that could have resulted in the payment of a premium when a premium was not 
necessary. 

3. Causes of Problems in the Identification of Evaluation Criteria 
Reasons provided by interviewees for the causes of problems with evaluation 
criteria are: 1) the nature of the PWS, 2) a lack of time to plan and construct the criteria, 3) 
a lack of training for both technical and contracting personnel, 4) a lack of communication 


between Government and industry and, 5) a lack of feedback conducted after the evaluation 


process. 








a. Nature of the Requirement 
A vague PWS is a major reason offered for difficulties in establishing 
evaluation criteria. If the PWS was well defined it would be easy to understand what is 
important in the performance of the tasks since the desired outcome is known. In the case 
of a vague PWS, the desired outcome is not so certain. Due to the dynamic nature of many 
programs, the requiring activity may have little idea of the tasks that will be required two 
years in the future. Therefore, it is difficult to select evaluation critena that will lead to the 
most advantageous procurement decision. 
b. Lack of Time 
Respondents to the survey who mentioned evaluation criteria as a problem 
generally believe that the requiring activity has the experience to select and tailor the correct 
evaluation criteria. However, the requiring activity does not always take the time to 
understand and really get at the heart of what is important in the performance of the contract 
and what criteria will really differentiate between offerors. Typically, evaluation criteria are 
selected at the requiring activity by the technical code requiring the services and forwarded 
to the contracting activity or department for approval. To save time in developing selection 
criteria, the requiring activity normally reviews the evaluation criteria for past and present 
contracts and utilizes the same cniteria for the current requirement. 
c. Training Deficiencies 
The lack of appropriate training for both technical and contracting personnel is 
another reason why poor evaluation criteria are written. Technical personnel are not always 
familiar with the entire source selection process and may not understand why evaluation 


criteria are so crucial in the proposal evaluation phase. This is especially true in the area of 


engineering and technical services where the contract period of performance may extend up 








to five years if all options are exercised. Due to this considerable length of time between 
evaluations, very few trained personne! from the previous selection process are available to 
relate the lessons learned from their previous experience. 

Technical personnel are also not always familiar with the different source 
selection methods used to procure services and how these methods relate to the 
development of evaluation cnteria. For example, technically acceptable low offeror source 
selections would emphasize responsibility criteria whereas "best value" source selections 
emphasize the qualitative criteria. Contracting personnel, familiar with different methods of 
source selection, generally lack the technical training required to understand what the 
requiring activity needs and therefore have difficulties in assisting the requiring activity 
develop the proper evaluation criteria. 

Training in the selection and design of evaluation criteria is normally provided 
by the contracting organization in the form of written guidance. Most of the contracting 
activities visited have provided detailed instructions on the different source selection 
methods with generic examples of criteria for the requiring activity to utilize. 

d. Communications among the Acquisition Planning Team 

Communications between members of the acquisition planning team consisting 
of technical and contracts personnel is critical, but not always sound. Technical personnel 
require an understanding of acquisition procedures and contracting personnel require an 
understanding of the technical aspects of the PWS. To assist in the communication flow, 
centralized contracting activities like NRCCs sometimes employ Requiring Activity 
Contracting Assistants (RACAs) who assist the requiring activity in the construction of a 
procurement package for NRCC procurement. Activities with technical and contracting 
departments at one location, have the advantage of proximity however, from this 


researcher's experience both centralized and decentralized organizations are susceptible to 








communication breakdowns between technical and acquisition personnel. This 
communication breakdown is normally attributed to the personal relationships between 
technical and acquisition personnel where both have preconceived notions about the other. 
For instance, some technical personnel may view contracting personnel as "clerks", always 
looking for the lowest price. Some contracting personnel may believe that technical 
personnel desire to circumvent the system whenever possible. The communication 
breakdown presents itself in the less than optimal design of evaluation criteria for "best 
value" evaluations. 
e. Communications with Potential Offerors 

Early communication with potential offerors is a method of receiving assistance 
in the preparation of evaluation criteria. Draft RFPs and presolicitation conferences are 
rarely used for engineering and technical services contracts because most procurement 
personnel believe that the process is not necessary and/or is too time consuming. The 
majority of personnel interviewed were of the opinion that draft RFPs and presolicitation 
conferences were worthwhile for a new procurement of complex services but not worth the 
effort for most engineering and technical services contracts even though "best value" 
evaluations were required. The following reasons support this opinion: 1) not all 
contractors attend presolicitation conferences or comment on draft RFPs, 2) regular RFPs 
elicit the same questions as draft RFPs and, 3) contractors already have a good 
understanding of engineenng and technical service RFPs and the evaluation criteria utilized. 
When considering the years of experience most contractors have with Government 
contracts and the emphasis on the use standardized RFPs at most activities, the assertions 


for not using draft RFPs and presolicitation conferences for all "best value" procurements 


are plausible. 











Proponents for the use of draft RFPs and presolicitation conferences for all 
"best value" procurements counter that not only do these tools assist in improving the 
quality of RFPs, but in the long run, improve the relationship between Government and 
industry. Regardless of the implications or the need, it was observed during this research 
that draft RFPs and presolicitation conferences were rarely used in "best value" 
procurements of engineering and technical services. 
f. Lack of Feedback 
One final cause attributed to improperly developed evaluation criteria, was a 
lack of feedback or lessons learned at the requiring activity after the evaluation. 
Specifically, very few interviewees admitted to seriously meeting after the evaluation to 
discuss the strong points or problems with the evaluation criteria. Many felt that a feedback 
session was a poor use of time since most of the members involved in the technical 
evaluation are tired of the process and very few of the personnel who participated in this 
evaluation will be present for the next one. The result, is an investment of time with limited 
learning for future procurements. 
§. Summary 
Identifying the proper evaluation criteria is crucial to the success of "best value" 
evaluations. However, according to the survey respondents, selecting the evaluation 
criteria which will differentiate between offerors and eventually lead to the most 
advantageous procurement decision is a difficult task which is not always successfully 
completed. Reasons for the problems in identifying adequate criteria occur due to the: 1) 
the nature of the PWS, 2) time for planning, 3) training for both contracting and technical 
personnel and 4) a failure in communication between contracting and technical personnel. 
In addition, draft RFPs and feedback sessions were noted as methods of assisting in the 


construction of criteria, both of which are rarely utilized. 











C. PROBLEM #2: THE TECHNICAL EVALUATION 


Technical evaluation personnel are not always educated concerning the importance of 
the technical evaluation to the "best value" source selection process. 

1. The Importance of Technical Evaluations 

The technical evaluation process is critical in determining which proposal offers the 

most advantageous procurement decision to the Government. An undisciplined technical 
evaluation can lead to the selection of an offeror who does not of ~ the "best value", or to 
the loss of a protest questioning the integrity of the source selection process. A strong 
technical evaluation is: 1) required to assist the SSA in deciding contract award, 2) 
necessary for discussions and debrief and, 3) a defense against protest. Problems with 
technical evaluations of proposals were mentioned by nearly half of the Government 
personnel interviewed. Many believe technical evaluations were a serious problem, "Our 
biggest problem with the greatest value method is the technical evaluators who want to 
quantify everything in numbers and not provide narrative". Others thought that problems 
with technical evaluations were relatively minor, "Correcting technical narrative is nothing 
more than a routine procedure similar to submitting correspondence through a supervisor 
prior to release". 


2. Deficiencies Mentioned by Contracting and Technical Personnel 
Regarding the Technical Evaluation Process 


Several comments collected during this research regarding the problems with the 
technical evaluation process are presented below. 
Technical evaluators grade too high. They rarely give failing grades. 
Being a member of a technical evaluation board is not looked upon highly by the 
senior engineers. The assignment to the technical panel needs to be recognized as an 


activity with value. Also, technical people see contracting personnel as clerks which 
does even less to help the process. 
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Technical evaluators are not well trained. They usually do it once in their career and 
do not want to take the responsibility. Contracting personnel can help by making the 
process (technical evaluation) as painless as possible. 


Technical evaluation committees are made up of technicians and engineers who are 
not sensitive to the (source selection) process. Strong preferences exist at each activity. 


Technical evaluations are bad- everywhere! The contracting officer needs to ensure 
the technical evaluators are trained to properly carry out their duties. 


Heads of requiring activities blame contracting officers because they are told how to 
conduct technica! evaluations and do not always believe they get the best possible 
contractor since they feel constrained by the rules. 

Like all other activities I know of, we too have a problem with technical narrative. 

There are normally lots of problems with the technical evaluation process. It is 
always an uphill battle. The technical evaluation panel is usually the 8th priority for any 
engineer. It is also very painful to get narrative comments from engineers; things that 
will live up to legal scrutiny that is. 

The technical evaluation committee did not work well together as a team. It took six 
months longer than scheduled to award the contract because of this. The narrative had 
to go back several times for rewrite. 


Technical evaluators see things in black and white. They like to use numbers when 
_ most of the differences in proposals cannot be equated to a numerical difference. 


The most common complaint concerning the technical evaluation was the lack of 
technical narrative which is required to document a "best value" award decision. As 
alluded to in the comments above, it appears that technical evaluators prefer the use of 
numerical grades over narrative to document the evaluation. This researcher has seen four 
examples at two activities of contract awards delayed for up to six months primarily due to 
submissions of poor narratives by the technical evaluation committees (TEC). The 
common scenano has the TEC submitting an evaluation to the contracting officer who feels 
that it is inadequate to support an award. The evaluation is sent back to the TEC, redone, 
and submitted back to the contracting officer. This may occur three or four times before a 
contract award is made. In one example involving two offerors for a five year $7 million | 
services contract, the technical evaluation was sent back to the TEC five times between 


February and July 1988. In between were two meetings and several conversations 





between the contract specialist and the requiring department. According to the contract 
Specialist, the procurement did attain the most advantageous procurement decision 
however, it was more costly than it should have been in terms of time and resources. 

In an extreme example, an offeror in a competition for engineering support 
services proposed to perform a PWS under a cost sharing arrangement with no fee in lieu 
of a CPFF contract. The contracting officer reasoned that the firm was trying to remain in 
business. This offeror proposed such a low price and favorable terms that on overall 
points the proposal was a clear winner. The second offeror, the incumbent, out scored the 
low priced offeror on technical points 52 to 43 although both were technically acceptable. 
The contracting officer admitted that he realized the requiring activity desired the incumbent 
offeror because of the higher technical score. Unfortunately, he could not, even after 
several meetings, extract the technical narrative necessary to make award to the incumbent 
offeror. As a result, the contracting officer had no justification but to award to the low 
priced technically acceptable offeror. One year into the contract, the requiring activity is not 
pleased with the contractor's performance and may elect not to exercise the next option and 
recompete the PWS. Although extreme, the above example illustrates what could happen 
without adequate narrative. 

3. Systems to Prevent Problems with the Technical Evaluation 

All of the activities interviewed have in place instructions and recommended 
procedures to train technical evaluators and facilitate the technical evaluation. NRCC, 
Philadelphia, has a four page instruction on the evaluation of offeror's technical proposals 
[Ref. 16]. Included with the instruction is a note from the Commanding Officer to the TEC 
which lists and describes common defects in technical evaluations. NRCC, Washington, 
D.C., in a package for all customer activities, not only includes instructions to the TEC, but 


also includes sample narratives justifying an evaluator's rating on an evaluation factor. 
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Naval Supply Center (NSC), Puget Sound, requires technical evaluators to pass a test on 
the source selection evaluation process prior to assignment to the TEC. Included in the test 
are 21 questions; some requiring explanations, while others are true and false. The test is 
presented in Appendix E. 

In addition to written instructions, technical evaluators are further instructed 
verbally prior to commencement of the actual technical evaluation. This instruction is 
delivered by either the contracting officer, his/her representative or the chairperson of the 
TEC. For centralized contracting activities like NRCCs, technical evaluators are 
encouraged to utilize conference space at the contracting activity to conduct their technical 
evaluations. This improves the process by sequestering the technical evaluators away from 
their day to day job requirements while dedicating their complete attention to the evaluation 
of technical proposals. It also places the TEC where they can receive immediate feedback 
from the contracting officer and legal counsei. 

4. Causes of Problems in Technical Evaluations 

With all of these systems in place, why do problems continue to exist between the 
contracting and technical personnel regarding the technical evaluation? The major reason is 
due to a lack of management commitment to the process. Management, on both sides, does 
not always comni't the resources considered necessary to ensure that an accurate and 
expeditious technical evaluation is completed. This can be attributed to a lack of personnel 
resources, time and an understanding of the importance of a disciplined evaluation. While 
discussing the importance of a disciplined evaluation, one contract specialist admitted that 
she prefers to have a protest on occasion, if only to have the management personnel at the 
technical activity understand the need for a disciplined technical evaluation process. 


One of the most frequently mentioned constraints on the technical evaluation 


process was that the engineers employed on technical evaluations have full time jobs in 








addition to the assignment on the technical panel. As a result, they can only dedicate 
random portions of their time to the evaluation. In addition, there exists a lack of incentive 
to become a member of the evaluation team. Normally, individuals are not evaluated nor 
do they receive performance awards based on their participation on a TEC. Therefore, 
there exists little motivation to expend great effort during the evaluation. The obvious 
motivation is in choosing the best contractor the requiring activity will have to work with 
for the next four or five years. Unfortunately, this is not always a superior motivating 
factor. In many instances, junior and new personnel are assigned to the TEC who have 
little understanding of the process. 

Contracting organizations can assist in the technical evaluation in a variety of 
ways. First, contracting officers and specialists can become familiar with the PWS to the 
point where they understand what the differences in technical proposals mean to successful 
completion of the PWS. This would assist the technical evaluation panel in verbalizing the 
differences in the technical narrative. Second, the contracting organization can be available 
and accessible at all times to answer questions from the TEC. Finally, contracting 
organizations can offer innovative training and feedback mechanisms like the test at NSC 
Puget Sound to improve technical evaluation procedures. 

§. Summary 

Technical evaluations are an important element of "best value" source selections. 
Without a disciplined technical evaluation and adequate documentation, it is difficult if not 
impossible to make the most advantageous procurement decision. Both contracting and 
technical personnel have issues concerning the technica] evaluation even though systems 
are in place to alleviate these issues. A lack of management commitment and understanding 
were the most frequently cited causes of problems with technical evaluations for 


engineering and technical services contracts. 
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D. PROBLEM #3: CONTRACTOR GAMING ON COST AND TECHNICAL 
PROPOSALS 


1. Gaming Defined 
The term "gaming" as used in this thesis, is defined as an innovation in cost or 
technical approach proposed by an offeror without adequate justification. These 
innovations are used to improve technical proposals and/or lower costs. Under gaming, 
the contractor has little or no intention of fulfilling the innovations proposed. Three 
reasons offered for gaming include: 1) the hope of becoming more competitive for award, 
2) a feeling of distrust in a system advertised to focus on technical over cost, and 3) to 
protect the firm from the Government, which is "notorious for being tough on customers”. 
This final view considers gaming as necessary in order to add a cushion to overhead costs 
to remain in business or generate an acceptable fee. 
2. Responses From Interviews on the Gaming Problem 
Several comments collected during this research which is indicative of the gaming 
problem are presented below. 

We get it (gaming) all the time. Typically we know where the problems exist and 
look at those areas. Gaming is used most often in overhead and skill mix. You also 
see a lot of gaming in other direct costs and escalation rates when they are not 
normalized. 

I do not believe we get the hours proposed in uncompensated overtime. 


Some contractors come in with great costs. However, when investigated more 
closely you note the costs do not match the technical approach. 


The problem we get with engineering support services is a contractor bidding the 
"A" team and you wind up getting the "C" team. 


"Best value" allows the technical people to use judgment. Personnel experience is 
usually the most important factor since sample tasks can be gamed by a good technical 
wnter. 


Many contractors submit reliable cost estimates; others do not. Cost realism to 
control gaming costs us lots of extra work, time and effort because you cannot rely on 
some contractors’ data. 





Contractors game the Government on both technical and cost. You should be 
suspicious when costs submitted by offerors are far apart. Contractors will tell you one 
thing in the technical proposal and the costs do not match up. 

There is a gaming problem with costs. Companies form joint ventures with other 
companies and nobody with resumes provided is assigned to the company being 
evaluated. Many companies setup new cost centers to bid on a proposal. 

Lots of companies play games with cost data after the contract is awarded. 

3. Where is Gaming Found? 

According to information obtained during this research, gaming is found in both 
cost and technical proposals. In the cost area, frequently mentioned gamed costs include 
direct labor rates and overhead proposed by the prime and subcontractors. Escalation rates 
and other direct charges are also gamed when not normalized. Areas mentioned where 
technical gaming occurs included: 1) changes in the proposed labor skill mix, 2) bait and 
switch tactics where the key personnei proposed are not delivered, and 3) technical 
approaches defined in sample tasks which are unworkable from the cost standpoint. In 
fact, an example of contractor gaming was related to this researcher in every factor and 
subfactor used in the evaluation of engineering and technical services contracts. 

After award, several interviewees mentioned task proposals from contractors with 
skill mixes which do not agree with the overall proposal skill mix awarded the contract. 
For instance, after award, a contractor may game by proposing a larger percentage of 
higher grade personnel than indicated in the onginal labor mix. The result is more 
overhead for the contractor and a higher rate of fund expenditure by the Government. This 
problem could be partly attributable to a poor estimate of the labor skill mix prepared by 
Government personnel or the inability of the COTR or ordering officer to negotiate a 


correct skill mix on the delivery or task order. For this reason, many consider cost 


overruns as an "inherent" problem in service contracting. 
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4. Examples of Contractor Gaming 

The following are examples of suspected contractor gaming found in the cost 
evaluation sections of two post business clearance memorandums examined during this 
research. Both clearances are for engineering and technical services and propose one base 
year with four option years. The evaluated costs for one of the contracts is approximately 
$16 million. The second contract has evaluated costs of approximately $30 million. 

In the first instance of suspected gaming, one of the contractors has submitted 
labor categories with a wide range of hourly rates. Shown below is a chart which reveals 
how the contractor distributed the hours within a given labor category among the various 
classifications that make up the labor category. The direct hourly rate is also shown for 
each classification and the percentage of effort for each classification is shown in the 


parenthesis. This example only includes two of the nine labor categones for illustration. 


Program Manager 


E18 250 hrs (10%) $33.38 

E8 2200 hrs (90%) 16.46 $18.19 
Equipment Specialist 

EL 250 hrs $12.51 

EJ 250 hrs 10.82 

ED 250 hrs 7.35 

EC 8250 hrs (92%) 6.85 $7.13 


As demonstrated above, and in six of the seven remaining labor categories, the majority of 
all hours proposed are at the low end of the pay scale. This in itself is not an indication of 
contractor gaming. However, a comparison to the composite rates of other offerors and the 
Government's estimation based on previous efforts indicates that the rates were not 
accurate, 1.e., the above rates are considerably lower. In addition, the comparison of 
hourly rates with the annual salaries displayed in another section of the proposal reflects 
wide disparities. "As an example, the gentleman proposed as the program manger has an 


annual salary of $83, 512 whereas the annual salary in the cost proposal is $37,835". 














In another example in the same evaluation, one of the contractors proposed 
overhead rates well below the forecasted rate for the next three years. Whereas the 
forecasted rate averaged 72%, the proposed rate averaged 59%. The justification offered 
by the contractor for the discrepancy was that they were establishing a new cost center for 
the effort. They offered three reasons for the the new cost center including: 

1. Type of work to be performed under the contract was different from existing 
contracts. 

2. The current division's rates would include costs the company feels should not be 
allocated to the contract. 

3. Existing division's rates are not competitive for this procureinent. 

DCAA felt that the above reasons did have merit, but were not sufficient for a 
number of technical and cost reasons. Since there was no cost information for the new 
proposed cost center, the Government utilized historic data and the orginal forecast to 
develop rates which turned out to be considerably higher than originally proposed by the 
contractor. 

In the final example, evaluated direct labor costs proposed by a subcontractor were 
increased by over 80% as a result of a cost realism analysis. The reason for the increase in 
evaluated costs is the dispanty between resume salaries presented in an appendix to the cost 
proposal and the hourly rates presented in the cost proposal. Once again the proposal 


average is well below other contractors’ proposals and the Government cost estimate based 


on previous experience. Table 3 was taken from the cost evaluation: 














TABLE 3 
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As evidenced in Table 3, a large disparity exists between the resume rates and the proposed 
rates. The evaluated cost was revised upward as a result. 7 
§. Uncompensated Overtime 
Uncompensated Overtime (UCOT) was mentioned as a gaming problem by a large 
number of Government and industry personnel. UCOT is defined in a DFARS change 


published in the Federal Register, Vol. 56, No. 172, page 43987, on September 5, 1991 as 


follows: 


The hours worked in excess of the normal 8 hours per day, 40 hours per week by 
professional employees who are exempt from the Fair Labor Standards Act without 
additional compensation. Compensated personnel absences, such as holidays, 
vacations, and sick leave, shall be included in the normal work week for purposes of 
computing uncompensated overtime hours. 

(The) "Uncompensated Overtime" rate is the rate which results from multiplying 
the hourly rate from a 40 hour work week by 40, and then dividing by the proposed 
hours per week. For example, 45 hours proposed on a 40-hour work week basis at 
$20.00 would be converted to an uncompensated overtime rate of $17.78 per hour. 
($20X40) divided by 45=$17.78. 


The DFARS change requires contracting officers to evaluate UCOT for technical 
risk in fulfilling contract requirements. Of particular note are: 
1. Unrealistically low labor rates or other costs that may result in quality or service 
shortfalls; and 
2. Unbalanced distribution of uncompensated overtime among skill levels and its use 
in key technical positions. 
This policy of evaluating the technical nsk of UCOT was tested in Comptroller General 
decision B-242020 of March 21, 1991 in the matter of Quantum Research, Inc. (QRI). A 
short summary of the decision follows: 

QRI bid on a Government contract in which it indicated that its professional 
employees work 2600 hours per year or 50 hours per week. (One industry standard is 
2080 hours annually, a 40 hour week.) The contracting agency downgraded QRI's 
proposal in part, to the use of such hours. GAO concluded that the downgrading was 


justified, because the 2600 hour work year created a risk regarding the stability of 
QRI's workforce. [Ref. 17:p. 8] 


Industry's position on UCOT is that its use has placed undue emphasis on price 


rather than quality in the source selection process for engineering and technical services. 





Industry believes the the Government should "level the playing field" by evaluating 
proposals for engineering and technical services on a "normalized" 40 hour per week basis 
regardless of the work week proposed or the acceptability of the accounting systems used 
by offerors. [Ref. 18:p. 3] 

While not endorsing the use of UCOT, the Government believes that disallowing 
its use “would be detrimental to the integrity of the Government's acquisition system to 
prohibit, restrict, or discourage any form of price competition, including the technique of 
proposing uncompensated overtime." [Ref. 18:p. 3] 

Interviews indicate that Government acquisition personnel recognize the problems 
with excessive UCOT and have instituted controls to evaluate the increased technical risk. 
In fact, all Government contracting organizations visited during this research claim to 
evaluate UCOT in light of increased technical risk. Several organizations review historic 
UCOT usage and do not permit increasing UCOT hours in new proposals. Even with 
these controls in place, this has not ended complaints by many Government and industry 
interviewees who claim they do not believe the Government is getting the UCOT hours 
proposed. Moreover, the extra time and resources dedicated to evaluating UCOT is not 
worth the effort. 

One senior Navy acquisition official believes that the problem with UCOT has "run 
its course” in that the controls instituted are sufficient to protect both Government and 
industry. Additionally, he believes the use of excessive UCOT is now on the decline. 
Judging from the number of comments received during this research from both 
Government and industry personnel, it is this researcher's belief that continued study of 


UCOT should be pursued to determine the costs and benefits of the Government's policy. 
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6. Why Gaming is a Problem 

Gaming is a problem for several reasons. First, gaming perturbates the budgeting 
process and can affect readiness. Activities budgeting funds based on a yearly proposed 
amount negotiated for an engineering and technical services contract are required to curtail, 
cancel or obtain additional funds to continue services once the budgeted funds are 
exhausted. Secondly, gaming increases the risks of not selecting the most advantageous 
contractor. Unwittingly, a SSA may select a contractor based on a superior technical 
proposal that the contractor can never deliver. The third reason gaming is a problem in 
"best value" evaluations is the amount of time and effort expended by the SSEB in an effort 
to combat it. All areas of the cost and technical proposals require detailed screening and 
comparison to avoid gaming. Finally, gaming increases the perception that contractors are 
unethical. Several interviewees mentioned ethical concerns with gaming, yet caveat their 
comments with the statement that the overwhelming majority of contractors were honest 
with regard to their proposals. 

7. Government Efforts to Combat Gaming 

The Government has developed several methods to combat contractor gaming in 
order to protect itself and responsible offerors. The first is through use of the cost realism 
analysis. As previously defined, the cost realism analysis is a review of the cost portion of 
an offeror's proposal to determine if the costs are realistic for the technical effort proposed. 
It is a technique which should be used where adequate price competition exists to guard 
against pricing which is unrealistically low and which may lead to quality deficiencies, 
performance shortfalls and/or cost overruns. The cost realism analysis allows the 
contracting officer to gather and analyze the minimum data needed to justify an offeror's 


proposal price. The cost realism analysis is an indicator of the offeror's understanding of 


the RFP's technical requirements. Costs proposed by the contractor are compared to 





historical data and the technica! proposal and increased if they are found to be unrealistic 
prior to making a decision on award. Appendix F contains a matrix similar to one used at 
many activities as a starting point for the cost realism analysis of support services contracts. 
The matrix is a rough order of magnitude companion of hours and costs between the 
existing contract and proposals received. 

The cost realism analysis is an effective method used to reduce contractor gaming. 
However, the analysis is often protested as arbitrary and incorrect. The protests occur 
because much of the cost realism analysis is judgmental in nature. Generally speaking, the 
Comptroller Generai will deny the protests if the contracting officer's analysis of the 
proposed costs is sufficient to make a supportable judgment that the proposed costs are 
realistic. This rationale is termed the "Informed judgment rule". 

The cost realism analysis 1s not performed for time and materials and indefinite 
delivery/ indefinite quantity contracts with fixed hourly rates since these rates will not be 
adjusted upward. Therefore, engineering and technical services awarded under these 
contract types with fixed hourly labor rates are evaluated under a cost reasonableness 
standard only. 

In addition to a cost realism analysis, another method of combatting contractor 
gaming is by establishing cost ceilings in areas where the Government suspects the use of 
unrealistic rates. Cost ceilings are negotiated to protect the Government. For instance, if a 
contractor proposes an overhead rate considered low by the contracting officer, a ceiling 
can be established requiring the contractor to remain below the overhead rate ceiling fora 


specified peniod of time. The problem with imposing ceilings is that a ceiling may defeat 


the purpose of awarding the contract to the most advantageous offeror in the first place. 











This is because, with a ceiling in place, the contractor is more apt to concentrate on cost in 
lieu of quality. According to one interviewee, imposing ceilings for a certain contract may 
mean higher charges under a different contract. 

Another method of mitigating gaming is through the use of normalization. 
Normalization involves measuring offerors against the same cost standard or baseline of 
circumstances where there are no logical differences in approach. This technique is 
typically applied in the cost areas identified as other direct costs and escalation rates. 
Normalization has the added benefit of reducing the time required for cost realism analysis 
since the elements normalized do not require additional investigation. 

Rejecting a proposal as non-reponsive if costs are unrealistic, or the overall 
proposal is unbalanced, is yet another method of combatting gaming. A typical statement, 
taken from a NAVAIR RFP to this effect reads: 

The Government may reject an offer as non-reponsive if it is materially unbalanced 
as to the prices for the basic requirement and the option quantities. An offer is 
unbalanced when it is based on prices significantly less than cost for some work and 
prices which are significantly overstated for other work. 

None of the contracting officers interviewed during onsite visits admitted to have eliminated 
offerors as non-reponsive based on unrealistic costs. In fact, many expressed concerns 
that just such an action is prohibitive due to the possibility of protests. 

8. Summary 

According to interviewees, contractor gaming is a comiaon practice when 
submitting proposals for engineering and technical services. Gaming in cost and technical 
areas can make an offeror more competitive for award. However, gaming can cause 
perturbations in the budgeting process and require a huge investment in time and resources 
by Government personnel to protect the integrity of the contracting system. The 
Government employs a vanity of techniques to combat gaming. Most are time consuming 


but necessary to ensure that the integrity of a "best value” evaluation process is kept intact. 
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E. PROBLEM #4: CONTRACTOR DISTRUST 

Contractors do not always trust "best value” evaluations. This claim is based on the 
perception that evaluation criteria communicated to the offerors through the RFP are not 
adequate to prepare a proposal fully responding to the Government's requirement and a 
belief that DoD contracting officers continue to make contract awa-ds based on the low bid. 


1. Comments Received Reflecting a Lack of Trust in "Best Value" 
Evaluations 


We (Government) provide everything to contractors in spite of their belief that we 
award on the basis of low bid. 


Industry is very competitive price wise. It is hard to say we do not like a contractor 
because his price 1s too low. 


I received a lot of high order criticism when briefing contractors about "best value" 
last month. They wanted to know the evaluation criteria and weights we use to rate 
them. 


Everyone needs to change their mindset before "best value" will happen. The 
current mindset is low bid. 


DoD dollars are shrinking and customers still say they want top of the line. 

- Contractors structure proposals in accordance with technical requirements because we 
say in the solicitation it is the most important factor. Later we award to the low cost 
offeror. No wonder contractors don't trust the system. 

Industry feels we are paying lip service to the "best value" concept. 
2. Systemic Areas for Distrust 
Problem #4 was heard from 15 Government and industry personnel who were 
interviewed duning this research. The views represented two areas inherent in "best value" 
evaluations that generate contractor distrust. Those areas are the communication of 
evaluation criteria and a lack of trust that the Government will award contracts on a "best 
value" basis in lieu of the low bid. 


a. Evaluation Criteria 


The Government policy of communicating evaluation criteria and their 


respective weights to offerors can cause a lack of trust between the Government and 











contractors. This is because the evaluation criteria used in "best value" evaluations to 
assess the efficiency and effectiveness of an offeror's approach are never completely or 
objectively exposed to offerors in the RFP. As a result, contractors claim they do not 
know what they are being evaluated on and are required to decide which elements of the 
proposal or PWS requires the most emphasis. Upon not being selected for contract award, 
unsuccessful offerors attribute the cause to not understanding the evaluation criteria. This 
can lead to a distrust of the "best value" evaluation process. 

In a low cost technically acceptable competition, an offeror is well aware that a 
contract award will be made solely on the low cost among those offerors rated technically 
acceptable. In a "best value" procurement, the contract award is based on a combination of 
factors. Even though section M of an RFP may state that the technical factor is considered 
to be more important than cost, this does not necessanly mean the high technical score 
always wins. This fact presents a difficult dilemma for contractors since a decision on 
where to place their emphasis, and how much, could make the difference between a 
winning and a losing proposal. "Best value” source selection forces offerors to ask 
themselves, "What should my proposed strategy be, high technical, low cost or something 
in between?" 

At the factor level, the same contractor dilemma exists. One Government 
interviewee stated that sometimes the TEC looks for "buzzwords" while evaluating 
proposals and other times the TEC looks for concepts. He further stated that evaluation 
criteria are generally held "close to the chest" in that they are not communicated to offerors 
in the RFP. For example, under the "Technical" factor, the subfactor "Technical 
Approach" is communicated in the RFP as being the most important relative to all the other 
"Technical" subfactors. "Technical Approach" in this situation requires that the offeror 


outline how he would accomplish a series of tests as delineated in the PWS. The subfactor 








does not inform the offeror of the the weights attached to each test nor does it inform the 


offeror of all the qualitative measures the Government will use to assess the effectiveness 
of the offeror's approach. For instance, will the Government provide a higher score to an 
offeror for accuracy above a certain process specification, and if so how much more? This 
type of information is not provided in the RFP and therefore, an offeror has to use 
judgment as to where to place emphasis. Upon not being selected for award, the contractor 
claims he did not understand the rules and thereafter loses faith in the process. 

Should evaluation criteria and their weights be communicated to offerors in the 
RFP of “best value" procurements? On the one hand, communicating the evaluation criteria 
and their weights in the RFP may eliminate feelings of distrust by providing offerors with 
objective measures of what the Government considers most important in an offeror's 
proposal. However, all Government interviewees posed this question believe that 
providing all criteria with weights may also stifle contractor innovation and limit flexibility 
of Government selection authorities who may then be forced into a mathematical 
formulation to determine the most advantageous procurement decision. 

b. "Best Value" Versus Low Cost 

Contractors do not always believe DoD is awarding contracts based on the 
"best value" concept. Instead, many contractors believe that DoD continues to place too 
much emphasis on the low bid. This concern was raised by numerous Government and 
contractor personnel interviewed during research. In essence, these interviewees believe 
the Government does not always "stay the course” in making "best value" contract awards 
but rather continues to favor low cost over quality in the selection of contracts for 
engineering and technical services. 

An example of this issue was discussed with the researcher by a senior Navy 


acquisition official who recalled a meeting with several contractors who were bidding on 
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Navy RFPs for engineering and technical services which advertised the payment of a 
premium for a superior technical proposal. Surprisingly, even with the offer of premiums, 
the contractors had decided upon a strategy of submitting the low bid in order to win the 
contracts. This strategy was based on the knowledge that defense cutbacks are limiting the 
funds available for support services contracts. The reasoning follows that with reduced 
funding available, even the advertisement of a premium will not stop the Government from 
awarding to the technically acceptable low offeror. Under this premise, the senior 
acquisition manager stated, "Contractors are reading the tea leaves of reduced funding for 
defense. We (Government) are operating like we still have plenty of money." The above 
example is a representation of the distrust many contractors harbor regarding “best value" 
evaluations in favor of low bid. 
2. An Opposing View 

Skeptical Government personnel believe that many contractors, who claim that 
they do not trust in “best value" source selections, are using "best value” as an excuse for 
poorly wnitten proposals which did not win the contract award. This line of reasoning 
shifts the blame for non-award to the Government and in particular, on the "best value" 
source selection process. Several Government interviewees expressed the opinion that 
contractors are very pleased with the "best value" concept when it leads to a contract award 
for their company. However, when it does not lead to a contract award, they claim that the 
process is flawed. 

3. Summary 

Contractor distrust of the "best value" source selection process can lead to gaming 
and protest. Both cost the Government in terms of time and resources. While considering 
the complaint of contractor distrust, a Government acquisition manager summed up the 


issue as follows: 





In some instances, all the words in the world will not make a contractor believe the 
Government is choosing the highest quality on a "best value" basis. Contractors have 
to see it in Our actions. This is a long term type thing although industry is crying to see 
results now. 


: F. CHAPTER SUMMARY 
Chapter IV provided a definition and analysis of the four most frequently mentioned 
problems identified during this research on the "best value" source selection process in the 
procurement of engineering and technical services. The chapter focused on: 1) identifying 
evaluation critena which discriminate between offerors, 2) the technical evaluation process, 
3) the contractor gaming of cost and technical proposals and 4) contractor distrust of the 
"best value" source selection process. 


Chapter V will offer conclusions and recommendations from this thesis research. 
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V. CONCLUSIONS AND RECOMMENDATIONS 


A. INTRODUCTION 
This chapter will discuss conclusions and recommendations reached as a result of this 
research. In addition, the research questions presented in Chapter ] are answered and 


recommendations for further research are presented. 


B. CONCLUSIONS 


1. The Expression "Best Value" is Defined in Terms of an Objective 
and a Process. 


The expression "best value” is defined both in terms of an objective and a process. 

As an objective, "best value" is defined as the most advantageous procurement 
decision. This derision can be based on low price, low overall cost or other factors. Other 
factors include technical approach, management capabilities and past performance. 

As a process, “best value" is defined in terms of a source selection, where all 
factors including cost are considered prior to making a decision on contract award. If 
properly conducted, the "best value” process should lead to the most advantageous 
procurement decision. To achieve optimum success, "best value" as a process relies on 
proper definition of the requirement, thorough acquisition planning and a disciplined 
technical evaluation. This conclusion is discussed in Chapter II. 


2. The “Best Value" Source Selection Process is Composed of 
Objective and Subjective Elements. 


Rating systems are objective tools employed to assist the SSA in making an award 


decision. Frequently though, the SSA is required to employ subjective judgments in 











deciding contract award. The need for subjeciive judgment in "best value" decisions can 


never be completely eliminated. This was discussed under definitions offered for "best 
value" in Chapter III. 

3. There are no Impediments Stated in Acquisition Regulations Against 
Utilizing "Best Value" Source Selection in the Procurement of 
Engineering and Technical Services. 

A review of policy guidance in Chapter I] revealed no regulations prohibiting the 
employment of a "best value” procurement Strategy. Current acquisition guidance is 
emphasizing “best value" as a method in which quality as well as price factors are 
considered in a contract award decision. "Best (greatest) value" is referenced in the FAR, 
in a recent change to the DFARS and was emphasized in a 22 March 1991 memo by the the 
Assistant Secretary of the Navy, the Honorable Gerald Cann. In addition, there are 
numerous agency and local instructions referring to the "best value" procurements. "Best 
value" procurements are not recommended for every item or service. In the area of 
engineering and technical services, "best value" procurements are reserved for requirements 
which dictate the complex integration of personnel for the accomplishment of complex 
PWSs. 


4. There is Little Standardization in the Rating Systems Utilized in the 
Navy to Evaluate Contractor Proposals. 


As discussed in Chapters II and III, Navy activities employ point systems, 
composite point systems with adjectival or color descriptors and strictly adjectival systems 
to evaluate contractor proposals under "best value" procurements. Although each system 
possesses advantages and disadvantages, each system is still able to obtain the most 
advantageous procurement decision. The common denominator in all rating systems with 
regard to "best value" lies in the judgment of the reasoned, and many times subjective 


decision of the SSA. 
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§. Procurement Personnel Know When to use "Best Value" 
Procurements for Engineering and Technical Services. 


In accordance with a variety of guidance in the FAR, DFARS, ASN (RDA) “best 
value" policy memorandum and local policy instructions, Navy procurement personnel of 
all grades consistently demonstrated that they understood when to use a "best value" 
procurement strategy. This was evidenced in responses received to Question #3 in Chapter 
Hil. 

6. Factors and Subfactors are Generally Standard in "Best Value" 
Procurements of Engineering and Technical Services. Evaluation 
Criteria Vary Depending on the Requirement. 

As dicussed in Question #4 of Chapter III, Technical, Management and Cost 
factors and related subfactors were generally standard in "best value" procurements for 
engineering and technical services throughout DoD. Evaluation criteria describing the 
factors are tailored to what is considered important in the requirement. Developing 
evaluation criteria which differentiate between offerors and Jead to the most advantageous 
procurement decision is a major issue in the acquisition planning process. 


7. Past Performance Data are Utilized by Most Activities, However a 
Standardized Format for Collection or Type of Data Does not Exist. 


The overwhelming majority of activities utilize past performance data as an 
evaluation criterion in “best value” procurements for engineering and technical services. 
However, many activities use qualitative measures such as improvements made to past 
deficiencies, while other activities utilize only objective measures of past performance such 
as labor costs, the skill mix utilized and uncompensated overtime hours used on past 
contracts. Objective measures involve experience in years, systems or both. Qualitative 
methods evaluate the effectiveness of the expenence. 

There also exist differences in how the data are collected. Many activities have 


formalized systems where questionnaires are mailed to all Government activities, who in 


the past, have utilized the the contractor for similar services. Other activities utilized less 
formal methods of telephone checks on past performance and the word of local technical 
personnel who have used the contractor in the past. 

There is no Navy standard for determining what past performance data are to be 
collected, how they are to be collected and how they are to be utilized. Vanous methods 
were encountered during this research. At present, there exists no centralized contractor 
rating system in DoD for professional services, although NAVSEA has instituted a 
Contractor Performance Index pilot program that appears to have promise. This conclusion 
refers back to data collected from Question#5 in Chapter III. 


8. Draft RFPs and Presolicitation Conferences are Rarely Utilized for 
"Best Value" Procurements of Engineering and Technical Services. 


Draft RFPs and presolicitation conferences are not required for every "best value" 
procurement of engineering and technical services. However, when any reasonable doubt 
exists as to the clarity of the PWS or the RFP, and time permits, then it is prudent to 
request comments on a procurement through the use of a draft RFP or a presolicitation 
conference. The basis for this conclusion was discussed under Problem #1 in Chapter IV. 


9. There is no Standard Guidance on how to Accomplish a Cost 
Realism Analysis for "Best Value" Procurements. 


As discussed in Chapter II and under Question #8 of Chapter III, acquisition 
regulations contain several definitions for cost realism and define when it should be 
employed, but there exists no standard guidance on the mechanics of performing a cost 
realism analysis for professional services. Currently, contracting activities perform many 
of the same steps used during cost analysis when cost ard pricing data are available. 
However, cost realism analysis is completed in a competitive environment and certified cost 


and pricing data are not available. 
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10. Although FAR 15.610 now Permits Awarding Without Discussions 
for "Best Value" Procurements, this Provision is Rarely Utilized. 


This researcher only encountered one "best value" procurement of engineering and 
technical services awarded without discussions. The award was made to the technically 
superior, low priced offeror and other offerors were not susceptible or capable of being 
made susceptible for award. However, a supervisory contracting officer at a NRCC 
estimated that now, up to 25% of all contract awards made at that particular NRCC are 
awarded on a "best value” basis without discussions. The past policy of awarding without 
discussions only to the low priced offeror was changed by Section 802 of the 1991 
National Defense Authorization Act and implemented in the FAR in April 1991. It is 
expected that the number of contract awards without discussions will continue to increase 
in the future as contracting personnel become familiar with the new regulation. 

11. Honest and Upfront Dialogue Between Requiring and Procurement 
Organizations is a Major Element to Successful "Best Value" 
Procurements. 

This conclusion was drawn repeatedly at every activity visited during the research. 
In order to achieve the most advantageous procurement decision through the use of “best 
value" procurements, a relationship of trust and respect must be fostered between the 
requiring and procurement organizations. A strong relationship improves "best value" 
procurements at every stage of the process from acquisition planning through contract 


award. 


C. RECOMMENDATIONS 


1. Contracting Activities Should Strive to Ensure That Their Customers 
Understand the Differences in Source Selection Methods Between 
"Best Value" Source Selection and Selecting the Low Cost 
Technically Acceptable Offeror. 


Requiring organizations play a major role in the selection of an acquisition 


strategy. By understanding the differences in source selection methods, requiring 





organizations who understand the technical requirements, are better equipped to make a 

. recommendation on the acquisition strategy. Depending upon the complexity of the 
services required and the evaluation criteria, either procurement method may lead to the 
most advantageous procurement decision. 

2. The Acquisition Planning Team and Technical Evaluation Committee 
Should Institute Feedback Mechanisms for the Improvement of 
Evaluation Criteria Used in "Best Value" Procurements. 

This recommendation would assist both technical and procurement organizations in 
the planning and evaluation of future acquisitions. Ideally, after award of a "best value" 
procurement, the acquisition planning team and the technical evaluators should meet to 
debrief the procurement. Specific areas to be addressed would focus on the evaluation 
factors, criteria, TEC composition and any area where uncertainties arose during the 
evaluation. A problem with this recommendation is the investment of time required to 

: conduct such a debriefing. However, if properly utilized, a debriefing of the procurement 
may save time and lead to more effective "best value" procurements in the future. 

3. Contracting Officers and Negotiators Should be Provided Abstracts 
of Comptroller General and BCA Decisions Germane to "Best Value" 
Procurements. 

NAVSUP should synopsize, or designate a field contracting activity to synopsize, 
Comptroller General and BCA decisions germane to "best value" procurements. These 
synopses should then be distributed to field contracting organizations for their information 
and education. This knowledge will help improve the "best value" source selection 


process. Synopsizing the decisions will ensure that the key issues in the cases are 


highlighted and save on the time required to read and comprehend a decision. Synopses 


should include only relevant facts of the cases without further interpretation. 








4. Emphasize Management Dialogue Between Contracting and Technical 
Organizations. 


This recommendation is basic to all contract‘ng and technical organizations and is 
presently followed in varying degrees. However, based on this researcher's findings, 
more effort is required. The technical and contracting organizations must emphasize the 
use of honest dialogue at all stages of a "best value" procurement. The key to success is in 
ensuring that each organization understands the requirements of the other. This is 
especially true in the areas of acquisition planning and technical evaluations of proposals. 
In the area of technical evaluations, it was mentioned in this thesis that the narrative in the 
technical evaluations are consistently inadequate. Continued dialog between the two 
activities could help correct this problem. 

5. The Customer Activity Must Dedicate Highly Experienced Personnel 

to the Technical Evaluation Committee or Risk Living with a 

Mediocre Service Contractor for Years in the Future. 

One of the major deficiencies noted in the technical evaluation process was a lack 
of qualified personnel to conduct the technical evaluation. The management at the requiring 
activity should ensure that experienced personnel are assigned to the technical evaluation 
committees. In addition, technical evaluators should be relieved of all other duties and be 
evaluated on their performance as a member of the technical evaluation committee. The 
management at the requiring activities should also seek innovative methods of assigning 
persons to the technical evaluation committee such as a jury duty type arrangement for a 
fixed period of time. 

Contracting organizations should recognize and be sensitive to the time and 
personnel constraints at the technical organizations. The contracting organization can assist 
by helping to streamline the acquisition process and be available for questions during the 
technical evaluation. Contracting organizations and technical organizations should also 


consider utilizing technical managers as the SSAs during procurements. 


6. The Navy Should Consider Implementing a Standardized Rating 
System for use in Evaluating Contractor Proposals. 


There exists a variety of rating systems used for "best value" procurements at 
contracting organizations throughout the Navy. Point systems, composite point systems 
with adjectival or color descriptors and strictly adjectival systems with no points are all 
currently used to rate contractor proposals for "best value" procurements. Standardizing a 
rating system throughout the Navy, may improve the efficiency and effectiveness of “best 
value" procurements in the long run because all concerned, Government and industry, will 
be familiar with the standard. On the other hand, a standard rating system may stifle 
innovative activities from experimenting with new systems which could be tailored for their 
needs. 


7. The Navy Should Provide Standardized Guidance on how to Perform 
a Cost Realism Analysis for Engineering and Technical Services. 


A regulation or instruction on how to perform cost realism analysis is not 
recommended. What is recommended is basic guidance similar to that available at NRCC 
San Diego, Long Beach Detachment which first defines cost realism and presents scenarios 
in line with past cost realism situations. The goal of this guidance should not be focused 
on “how to" but rather to sensitize the cost and technical evaluators to cost realism issues. 


8. Key Personnel Salaries and Other Cost Information Should be 
Integrated into Technical Evaluations. 


Salary and other cost information relevant to the technical proposal should be 
integrated into the technical evaluation to help ensure that the costs are realistic, protecting 
both the Government and responsible contractors from unrealistic rates. This information 
should be compared to the level of experience of the individuals and the technical approach 
proposed. Problems with maintaining the privacy of contractor personnel proposed could 


be averted by providing a salary range versus a fixed figure. 
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9. Contracting Organizations Should Streamline the "Best Value" 
Procurement Process by Emphasizing the use of Award Without 
Discussions. 


Making an award without discussions permits the contracting activity to streamline 
the source selection process by eliminating the requirement for discussions and subsequent 
BAFOs. The key to awarding without discussions begins with good acquisition planning 
and continues through a disciplined evaluation process. Making an award without the 
requirement for discussions should be one of the foremost objectives of the contracting 
organization since it can significantly streamline the process. Thoughtful scenarios should 
be developed and discussed at each activity as to when awarding without discussions on a 
"best value" basis would be applicable. 

10. Contracting Activities Should Institute Formalized Methods of 
Collecting and Evaluating Qualitative Past Performance Data for use 
in "Best Value" Procurements. 

Contractors need to understand that the quality of past contract performance will be 
one of the evaluation criteria for all "best value" procurements. One method of 
disseminating this policy to contractors, is to implement formalized procedures to collect 
and evaluate past performance data. As mentioned in the conclusions, there exists a variety 
of collection methods and types of data utilized in the area of past performance evaluations. 
A standardized Navy policy is not recommended. However, formalized approaches 
stressing qualitative assessments of past performance appear to have the most promise in 


ensuring contractors understand the importance of quality contract performance. 


D. ANSWERS TO RESEARCH QUESTIONS 
1. Primary Research Question: What is the Most Efficient and Effective 
Method of Implementing a "Best Value" Procurement Strategy for 
Engineering and Technical Services Contracts? 
There is no one specific method for implementing "best value" that is most efficient 


or effective. However, there are basic tenets to consider in every “best value" 





procurement. “Best value" procurements are accomplished within a framework of: 1) 
properly identifying the requirement, 2) preparing clear and relevant evaluation criteria, 3) 
conducting a disciplined evaluation process and, 4) determining the most advantageous 
procurement decision to the Government. This framework requires continuous planning 
and evaluation at every stage to improve the efficiency and effectiveness of the process. 
This research highlighted specific areas for contracting organizations to review prior to 

and during "best value" procurements which include the: 

1. Training and communications between the members of the acquisition planning 

organization. 

2. Need for the use of draft RFPs and presolicitation conferences to solicit industry 
input. 
Rating systems used by the organization. 
Training and motivation of the technical evaluation committee. 


The cost realism analysis. 
Debriefing or lessons learned process after contract award. 


Sat Le aaa 


2. Subsidiary Question 1: What is "Best Value"? 

For purposes of this research, "best value" was defined in terms of an objective 
and a process. As an objective, "best value" represents the most advantageous 
procurement decision available to the Government. This decision is based on low price, 
low overall cost or other factors. "Best value" defined as an objective has extremely broad 
applications. One can obtain the "best value" under any source selection method as long as 
the product or service meets the criteria of providing optimum satisfaction of mission 
needs. 

"Best value” is also defined in terms of a source selection evaluation process where 
all relevant factors are taken under consideration prior to making an award decision. The 
end result of this process represents the most advantageous procurement decision available. 
In this context, procurement decisions are not based solely on price. 

As a process, "best value" directly contrasts with technically acceptable low offeror 


source selections which make contract awards based on the low price among technically 
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acceptable offerors. "Best value" emphasizes other factors in addition to price and 


establishes different levels of technical merit from which to make the most advantageous 
procurement decision. 


3. Subsidiary Question 2: Under What Circumstances is "Best Value" 
Employed? 


"Best value" is generally reserved for the procurement of complex goods and 
services where alternate approaches for satisfying a requirement are available. This is due 
to the often times subjective nature of a "best value" procurement where evaluation factors 
and criteria are utilized to discriminate between solutions offered by competing proposals. 
As a result, requirements necessitating the use of performance specifications or complex 
services are typically considered for "best value" procurements. Conversely, products with 
detailed specifications and routine services are acquired based on low cost. 

4. Subsidiary Question 3: What Source Selection Methods are Used for 
the "Best Value" Procurement of Engineering and Technical 
Services? 

Two source selection methods are typically used for "best value" procurements of 
engineering and technical services. Those methods are formal source selection and 
streamlined or informal source selection. Both methods utiiize evaluation factors to 
differentiate between offerors. Formal source selection is administratively expensive and is 
generally reserved for complex or higher cost requirements. The streamlined source 
selection method is more common in practice. Streamlining is normally accomplished by: 
1) limiting the number of pages in a proposal, 2) reducing the number of evaluation factors 


and/or 3) reducing the number of personnel involved in proposal evaluation. 








§. Subsidiary Question 4: What Factors and Subfactors are Utilized in 
"Best Value" Procurements for Engineering and Technical Services? 


From a sample taken during this research, evaluation factors and subfactors used 
in "best value” evaluations are fairly standard in title and content. The factors used include 
technical, management and cost. Technical and cost were factors in all "best value" 
procurements for engineering and technical services. Management was primarily listed as a 
subfactor of technical. Subfactors under the above factors include personnel experience, 
technical approach, management plan, manpower utilization matrix, sample tasks and past 
performance. 

6. Subsidiary Question 5: What are the Problems or Impediments to 
Implementing "Best Value" Source Selection for Engineering and 
Technical Services? 

Several problems and impediments were identified during the course of this 
research. The majority of the problems and impediments were in the areas of acquisition 
planning and proposal evaluation. Specific problems mentioned most often during 
interviews included; 1) the identification of evaluation criteria which will differentiate 
between offerors and lead to the most advantageous procurement decision, 2) the technical 
evaluation process, 3) contractor gaming of cost and technical proposals and, 4) contractor 
distrust of the “best value" procurements which advertise technical ability over cost. 
Impediments to correcting the problems pervasively centered on a lack of time, training, 
communications and resources necessary to plan, document and conduc. a "best value" 
procurement. 


7. Subsidiary Question 6: How do Activities Quantify Paying a 
Premium on "Best Value" Procurements? 


All activities questioned emphasized reasoned judgment prior to extending a 
premium to a contractor for a superior technical proposal. However, most activities also 
conduct a cost benefit analysis in which additional technical points are equated to additional 


dollars. The cost benefit analysis is carried out by the technical evaluators, who at certain 
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activities, are provided the cost differentials. Other activities do not supply the technical 


evaluators with a cost differential. In these cases technical evaluators are provided with the 
differences in technical proposals and are asked to justify the differences without the dollar 


differential at hand. 


E. RECOMMENDATIONS FOR FURTHER RESEARCH 

1. Conduct a cost benefit analysis of uncompensated overtime use for engineering 
and technical services at one activity. 

2. Investigate the most efficient and effective method to collect and evaluate past 
performance data for use in the source selection of professional services contracts. 

3. Determine the elements for and construct a "guide" on the conduct of a cost realism 
analysis for professional services procurement at NA VSUP field activities. 

4. Evaluate the technical proposal evaluation process for engineering and technical 
services at two field activities. Compare and contrast the results. 

5. Study the Acquisition Review Program (ARP) at NWC China Lake, California and 
make recommendations for improvement and export to other activities. 

6. Define the applicability of making contract awards without discussions in "best 
value" procurements. Present a background and scenarios where awarding without 
discussions may streamline the "best value" procurement process. 

7. Investigate rating systems used in "best value" procurements and decide whether 


the Navy should implement a standardized rating system. 











APPENDIX A 


THE ASSISTANT SECRETARY OF THE NAVY 
(Research, Development and Acquisition) 
WASHINGTON, D.C. 20350-1000 


MAR 22 1991 


MEMORANDUM FOR: PROGRAM EXECUTIVE OFFICERS 
DIRECT REPORTING PROGRAM MANAGERS 
COMMANDERS OF SYSTEMS COMMANDS 
DEPUTY CHIEF OF STAFF FOR INSTALLATIONS AND 
LOGISTICS, HEADQUARTERS, U.S. MARINE CORPS 
COMMANDING GENERAL, MARINE CORPS 
RESEARCH, DEVELOPMENT AND ACQUISITION 
COMMAND 
COMMANDER, MILITARY SEALIFT COMMAND 
COMMANDING OFFICER, AUTOMATIC DATA 
PROCESSING SELECTION OFFICE 
COMMANDING OFFICER, NAVAL COMMERCIAL 
COMMUNICATIONS CENTER 


Subj: "BEST VALUE" CONTRACTING POLICY (DCN 18210021/APIA-01-0001) 


The purpose of this memorandum is to reemphasize the Navy's commitment to 
competitively select sources on the basis of the "best value” to the Government. In 
determining factors to be evaluated other than price, the nature of the supplies or services to 
be acquired must be considered. 


Factors impacting "best value" decisions include technical competence, proven past 
performance, management capability, life cycle costs and quality. Evaluation of these 
factors should be structured to ensure that they are given due consideration in determining 
the overall benefit associated with the offered price. 


"Best value” evaluations should be employed wherever possible and especially in 
weapons system, federal information processing (FIP) and professional and technical 
service support acquisitions. Requirements which dictate complex integration of people, 
equipment, hardware, innovation and software should also be evaluated on a "best value" 
basis. However, when non-complex, routine requirements are being procured and the 
product to be delivered is clearly defined at the outset of the procurement, it may be 
appropriate to award to the lowest priced, technically acceptable offeror. 


Please circulate this memorandum to the appropriate personnel as it reemphasizes the 
Navy's commitment to "best value” when acquiring systems, products and services for the 
Fleet. 


Gerald A. Cann 
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APPENDIX B 


LIST OF INTERVIEWEES 


The following is a list of persons who directly provided information for this research. 
Interviews consisted of both telephone conversations and personal visits. 


1) Ayars, J., Deputy Director of Contracts, Naval Supply Center, Puget Sound, WA, 
July 30, 1991. 


2) Barker, W., Director Information Resources Contracts Division, Naval Regional 
Contracting Center, Washington, DC, September 11, 1991. 


3) Bugby, D., Director oi Business Government Policy, Professional Services 
Council, Washington, DC, August 1, 1991. 


. 4) Camph, B., Manager of Contracts, SRI International, Menlo Park, CA, various, 
September 1991. 


5) Canales, M., Contracting Officer, PMA Services, Naval Air Systems Command, 
Washington, DC, September 11, 1991. 


6) Canfield, T., Capt., SC, USN, Director of Command Inspection Division, Navy 
Inspector General Staff, Washington, DC, July 28, 1991. 


7) Carberry, V. S., Executive Director of Contracts Managemeni, Naval Air Systems 
Command, Washington, DC, September 11, 1991. 


8) Celloto, D., Deputy Counsel, Naval Regional Contracting Center, Philadelphia, 
PA, August 2, 1991. 


9) Cengia, R., Supervisory Contracting Officer, Naval Regional Contracting Center, 
San Diego Detachment, Long Beach, CA, various, July - November, 1991. 


10) Cook, O., Contracting Officer, CTA Systems Support Contract, China Lake, CA, 
August 13, 1991. 


11) Finnigan, M., Manager, Services Contract Division, Naval Aviation Center, 
Indianapolis, IN, August 7, 1991. 


12) Ford, F., Office of Assistant Secretary of the Navy, Washington, DC, various, 
August - October, 1991. 


13) Hallum, L., Contract Cost/Price Analyst, Directorate of Contracting, Fort Ord, 
Monterey, CA, various, July - August, 1991. 











14) Hart, N., Cdr., SC, USN, Director of Contracts, Naval Supply Center, Puget 
Sound, WA, July 27, 1991. 


15) Hauf, B., Capt., USAF, Air Force Logistics Management Center, Gunter Air 
Force Base, AL, September 25, 1991. 


16) Hauenstein, W., Radm., SC, USN, Competition Advocate of the Navy, Office of 
the Assistance Secretary of the Navy, Washington, DC., August 1, 1991. 


17) John, W., Division Director, Missile and Systems Contracts Division, Naval Air 
Systems Command, Washington, DC, September 11, 1991. 


' 18) Kaczmarek, T., Senior Contract Negotiator, Naval Regional Contracting Center, 
Philadelphia, PA, July 18, 1991. 


19) Kalapos, M., Capt., SC, USN, Director of Contracts, Naval Training Systems 
Center, Orlando, FL, July 27, 1991. 


20) Kaufman, S., Deputy Associate Administrator, Office of Federal Procurement 
Policy, Washington, DC, August 8, 1991. 


21) Kolis, T., Procurement Analyst, National Aeronautics and Space Administration, 
Ames, CA, July 10, 1991. 


22) LaFollette, C., Chief, Management Branch for Engineering and Technical 
Services, National Aeronautics and Space Administration, Ames, CA, various, 
August - September, 1991. 


23) Lavering, F., Program Manager, ADP and Telecommunications Support Systems, 
Boeing Computer Services, China Lake, CA, August 13, 1991. 


24) Manson, B., Deputy Director of Contracts, Naval Weapons Center, China Lake, 
CA, various, August - November, 1991. 


, 25) McCabe, T., Chief, Contract Division, Defense Nuclear Agency, Alexandria, VA, 
August 28, 1991. 


26) McCarthy, P., Ledr, SC, USN, Contracts Section Head, Naval Sea Systems 
Command, Washington, DC, various, July - October 1991. 


27) Monckton, W., Contract Specialist, Naval Weapons Center, China Lake, CA, 
August 14, 1991. 


28) Moris, M., Chief of Contracting Section, Department of the Army Central 
Contracting Office, Atlanta, GA, July 12, 1991. 


29) Murphy, B., Head, Radar and Fire Control Branch, Contracts Directorate, Naval 
Sea Systems Command, Washington, DC, July 17, 1991. 


30) Norman, R. Vice President of Contracts and Administration, ROH Inc., 
Alexandna, VA, July 20, 1991. 











31) Passar, A., Maj. USA, Contract Law Attomey, Department of the Army Central 
Contracting Office, Atlanta, GA, July 30, 1991. 


32) Patterson, J., Contract Specialist, Naval Weapons Center, China Lake, CA, 
August, 12, 1991. 


33) Paul, D., Contract Specialist, Naval Weapons Center, China Lake, CA, August 15 
1991. 


34) Penn, A., COTR, Boeing ADP Support Services Contract, Naval Weapons 
Center, China Lake, CA, Augusi 13, 1991. 


35) Reese, B., Director of Contracts, Techmatics Corporation, Alexandria, VA, July 
27, 1991. 


36) Richter, R., Assistant for Engineering Programs, Naval Ship Weapon Systems 
Engineering Station, Port Hueneme, CA, July 17, 1991. 


37) Rainwater, S., Ordering Officer, Boeing ADP Support Services Contract, Naval 
Weapons Center, China Lake, CA, August 14-15, 199}. 


38) Riggs, R., Associate Department Head, Engineering Department, Naval Weapons 
Center, China Lake, CA, August 14, 1991. 


_ 39) Rodgers, D. Supervisory Contracting Officer, Naval Regional Contracting Center, 
San Diego Detachment, Long Beach, CA, June 25, 1991. 


40) Rumberger, R., Branch Head, Systems Control Branch, Naval Air Systems 
Command, Washington, DC, September 12, 1991. 


41) Ryan, J. Cdr., SC, USN, Deputy Division Director, Naval Air Systems 
Command, Washington, DC., various, August - November, 1991. 


42) Siminkski, M., Chief of Contract Operations Directorate, Vandenburg Air Force 
Base, CA, August 29, 1991. 


43) Sueur, R., Cdr., SC, USN, Commanding Officer, Naval Regional Contracting 
Center, San Diego Detachment, Long Beach, CA, June 24, 1991. 


44) Safford, D., Legal Counsel, Naval Regional Contracting Center, San Diego 
Detachment, Long Beach, CA, June 25, 1991. 


45) Swizewski, J., Supervisory Contract Negotiator, Naval Regional Contracting 
Center, Philadelphia, PA, August 2, 1991. 


46) Tarves, A., Supervisory Contract Negotiator, Naval Regional Contracting Center, 
Philadelphia, PA, July 30, 1991. 


47) Taylor, S., Supervisory Contracting Officer, Naval Regional Contracting Center, 
Washington, DC, September 11, 1991. 
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48) Trunk, M., Procurement Policy Analyst, Naval Supply Systems Command, 
Washington, DC, various, July - October 1991. 


49) Utendorff, D., Technical Assistant to the Deputy Chief of Staff for Contracting, 
Aeronautical Systems Division, Wright Patterson Air Force Base, Dayton, OH, 
August 28, 1991. 

50) Vollmer, J., Project Director, EGG Inc., Alexandria, VA, July 17, 1991. 


51) Webb, D., Contracting Officer, Naval Regional Contracting Center, San Diego 
Detachment, Long Beach, CA, June 25, 1991. 


52) Wice, M., Ordering Officer, COMARCO Support Services Contract, Naval 
Weapons Center, China Lake, CA, August 11, 1991, 


53) Williams, C., Deputy Director, Onsite Field Support Office, Naval Supply 
Systems Command, Washington, DC, July 22, 1991. 
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APPENDIX C 


SURVEY QUESTIONNAIRE 


Government: 


1. 


2. 


How would you define "best (greatest) value?" 


Does your activity use numerical scoring, adjective grades or colors to ev~'uate 
proposals to achieve “best value"? Why is that rating system used? 


3. How do you decide a whether certain requirement requires the usage of the "best 
value" method? 

4. What are the factors and subfactors your activity uses for "best value" procurements 
of engineering and technical services contracts? 

5. How is past performance data collected and utilized in “best value" source selection 
for engineering and technical services contracts? 

6. What are some of the problems or issues your activity faces with "best value" 
procurements of engineering and technical services contracts? 

7. How does your activity determine the need to pay a premium for a technically 
superior proposal over a proposal which is technically acceptable, yet lower priced? 

8. How does your activity conduct cost realism analysis for engineering and technical 
services contracts? 

Contractor: 

1. How would you define "best (greatest) value?" 

2. How do you feel about the "best value" concept utilized by the Government in 
awarding contracts? 

3. Does your firm have any problems with "best value" solicitations let by DoD for 
engineering and technical services? If so, what are they? 

4. How do you feel about awarding contracts on a "best value" basis without 
discussions? 

5. In your view, does the Government do an adequate job of cost realism analysis? 
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APPENDIX D 


CONTRACTOR EVALUATION QUESTIONNAIRE 


Date: 

Contract #: CONTRACTOR: 
Option period being evaluated: 
COTR Name: Code: Phone: 
The Rating System is as Follows: 

O - Outstanding M - Marginal 

B - Better U - Unacceptable 

A - Acceptable 
Technical Approach 
Did the contractor perform as required? O B A M U 
How well did the contractor continually improve his 
approach in solving technical problems? O B A M U 
Schedule 
How well did the contractor make delivery of CDRL 
items in accordance with the terms and conditions? 
Was rework required? O B A M U 
How well did the contractor demonstrate organizational 
methods to recognize possible delays and take corrective 
actions immediately? O B A M U 


How well did the contractor keep the Government 
adequately informed of schedule issues. O B A M U 


Management Approach 


How well did the contractor maintain an open line of 
communication so that the requiring code was apprised 
of technical, cost and schedule issues? O B A M U 





How well did the contractor show flexibility and 
ingenuity in problem solving? O B A M U 


How well did the contractor provide clear, concise 

and informative written reports? O B A M U 
Cost 

How well did the contractor provide/recommend a 


reasonable labor mix? O B A M U 


Personnel 
Did the contractor have a stable work force? O B A M U 


Did the contractor provide a quality work force? O B A M 
Endorsement 


Based on the experience you have had with this contractor, would you elect to have them 
perform a contract for you again? 


Please indicate along the scale your overall rating of the 
contractor's performance under this contract? O B A M U 


General_Information 


What was the contractor's greatest strength? 


What was the contractor's greatest weakness? 


== lla eS SS SS SS SS SS SS SS SS SS SS SS Se 


PR# will be placed under the new 
option year 


Estimated Hours 


Estimated Price 
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APPENDIX E 
NSC PUGET SOUND TECHNICAL EVALUATOR TEST 


1. Under what circumstances should Source Selection Procedures be used? 
2. Who develops the evaluation plan, and once developed, what is the next step? 


3. The plan must include six major factors. What are they, and discuss/describe each 


4. What are the duties of the Evaluation Committee Chairperson? 

5. Who should be on the Evaluation Committee, and what are their functions? 

6. How are the Source Selection plan criteria used by the PCO in the solicitation? 

7. Evaluation criteria must be in concert with the statement of work. The criteria must 
also possess certain other characteristics. Indicate whether the following statements are 
true or false. 


a. Criteria should overlap each other as littles as possible. 


b. You should develop as many criteria as possible so that scoring and ranking can 
be broken down as much as possible. 


c. You should include criteria that all offerors are tikely to meet so that only 
formulas or definitive rules can be used to evaluate, and to ensure that all offerors will 
receive some points. 


d. Cniteria should not be so narrowly defined that evaluation of responses becomes 
a yes/no decision. 


e. Criteria should be worded as generally as possible to allow various 
interpretations of the evaluation plan in order that varied responses will be received from 
offerors. 


f. Criteria should challenge the offeror to describe a technical approach in sufficient 
detail to that meaningful differences among offerors will be brought out through the 
evaluation process. 


8. The plan may include mandatory or "non-negotiable" standards. What are some of 
the dangers of using these, and how must they be used? 


9. In the evaluation process, may offerors be compared to each other? Discuss your 
answer. 
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10. The most commonly used approach to ranking/rating proposals is the 
narrative/numerical methodology. Describe the process. 


11. In this example, the Source Selection Plan contains a technical factor of Experience 
for evaluation. Can only the technical proposal submitted be evaluated and considered? 
Can the experience indicated be verified/researched? 


Why/Why not? 


12. Who has the ultimate authority to approve the Source Selection Plan and determine 
which offeror will be awarded the contract? 


13. True or False: The offeror receiving the highest overall (cost and technical) evaluated 
score automatically is selected for award and additional evaluation or justification is not 
required. 


Why/why not? 
14. Explain what is meant by offerors being technically equivalent. 


15. When two or more offerors receive essentially the same technical scores, are they 
always technically equivalent? 


Why/Why not? 


16. What must be done if individual committee members’ technical scores of an offeror's 
proposal vary widely on one or more technical factors? How wil! the Committee's 
summary and final recommendation be affected? 


17. The Source Selection Plan will establish technical factors to be evaluated and their 
weights. Should the plan and/or the committee subdivide by weights each technical factor? 


Why/Why not? 


18. The plan will contain a technical evaluation rating standard (usually numeric) defining 
the ratings 10 through 0. How will the rating standard be used during the technical 
evaluation? Also, must the numeric rating assigned by supported by written 
documentation? Explain your answer. 


19. Assume the technical evaluation will be on four technical factors to be evaluated and 
the offeror being evaluated scores a perfect score in three of the four factors but scores a 
one or zero on the fourth. As a result, the overall numeric technical score indicates a 
satisfactory rating. How should the committee's evaluation and final recommendation 
address this situation? 


20. Explain the relationship between the evaluation criteria set forth in the Source 
Selection Plan and the technical proposal submission requirement set forth in the 
solicitation. 


21. At the conclusion of technical evaluations, what documents must be submitted to the 
PCO by the Committee Chairman. 
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